 |
TensorRT
7.2.1.6
NVIDIA TensorRT
|
Looking for a C++ dev who knows TensorRT? I'm looking for work. Hire me!
|
|
7.2.1 - 2020-10-20
Added
- Polygraphy v0.20.13 - Deep Learning Inference Prototyping and Debugging Toolkit
- PyTorch-Quantization Toolkit v2.0.0
- Updated BERT plugins for variable sequence length inputs
- Optimized kernels for sequence lengths of 64 and 96 added
- Added Tacotron2 + Waveglow TTS demo #677
- Re-enable
GridAnchorRect_TRT plugin with rectangular feature maps #679
- Update batchedNMS plugin to IPluginV2DynamicExt interface #738
- Support 3D inputs in InstanceNormalization plugin #745
- Added this CHANGELOG.md
Changed
- ONNX GraphSurgeon - v0.2.7 with bugfixes, new examples.
- demo/BERT bugfixes for Jetson Xavier
- Updated build Dockerfile to cuda-11.1
- Updated ClangFormat style specification according to TensorRT coding guidelines
Removed
bool tf32
Definition: sampleOptions.h:112
Plugin factory used to configure plugins with added support for TRT versioning.
Definition: NvCaffeParser.h:116
EnqueueFunction mEnqueue
Definition: sampleInference.cpp:460
xy_t()
Definition: maskRCNNKernels.h:260
def chpt_to_dict_arrays_simple(file_name)
Definition: chptToBin.py:30
const char * getPluginVersion() const override
Return the plugin version.
Definition: batchTilePlugin.cpp:156
std::map< std::string, nvinfer1::Weights > loadWeights(const std::string &file)
Loads weights from weights file.
Definition: sampleMNISTAPI.cpp:389
def __init__(self, path)
Definition: backend/tf/loader.py:112
def _tensor_quant(inputs, amax, num_bits=8, unsigned=False, narrow_range=True)
Definition: tensor_quant.py:318
uint32_t scale_softmax
Definition: fused_multihead_attention_v2.h:58
virtual int32_t enqueue(const PluginTensorDesc *inputDesc, const PluginTensorDesc *outputDesc, const void *const *inputs, void *const *outputs, void *workspace, cudaStream_t stream)=0
Execute the layer.
bool teardown()
Cleans up any state created in the sample class.
Definition: sampleMLP.cpp:349
size_t mAnchorSizeNum
Definition: proposalPlugin.h:120
Definition: frcnnUtils.h:51
input_tensor
Definition: test_ir.py:136
inputs
Definition: isolate.py:54
static half_float::half infinity()
Positive infinity.
Definition: samples/common/half.h:4246
Entropy calibrator 2.
Definition: NvInfer.h:6078
void serialize(void *buffer) const override
Serialize the layer.
Definition: batchTilePlugin.cpp:113
LoopOutput
Enum that describes kinds of loop outputs.
Definition: NvInfer.h:4487
uint64_t hashID(Data_type type, unsigned int sm) const
Definition: fused_multihead_attention.h:322
unsigned char fused_multihead_attention_int8_384_64_kernel_sm80_cu_o[]
Definition: fused_multihead_attention_int8_384_64_kernel.sm80.cpp:20
ComponentWeights::ptr mWeights
Definition: slpEmbedder.h:51
def __init__(self, model)
Definition: backend/onnx/loader.py:179
calculates attention vector from context and decoder output vectors
Definition: attention.h:32
n_early_size
Definition: demo/Tacotron2/waveglow/model.py:168
def init_decoder_tensors(decoder_inputs, decoder_outputs)
Definition: inference_trt.py:118
IErrorRecorder::RefCount decRefCount() noexcept final
Decrements the refcount for the current ErrorRecorder.
Definition: ErrorRecorder.h:105
engine
Definition: backend/trt_legacy.py:270
std::basic_istream< charT, traits > & operator>>(std::basic_istream< charT, traits > &in, half &arg)
Input operator.
Definition: ieee_half.h:2218
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleOnnxMNIST.cpp:49
use_qat
Definition: builder.py:104
void setShouldLog(bool shouldLog)
Definition: pluginLogging.h:92
int numClasses
Definition: maskRCNNKernels.h:86
Convenience object used to initialize LogStreamConsumerBuffer before std::ostream in LogStreamConsume...
Definition: pluginLogging.h:107
def load_network(builder, network, parser)
Definition: Polygraphy/examples/api/03_interoperating_with_tensorrt/example.py:39
std::string GRID_ANCHOR_PLUGIN_NAMES[]
Definition: gridAnchorPlugin.cpp:29
std::vector< uint8_t > buffer
Definition: samples/common/common.h:664
constexpr const char * INSTANCE_PLUGIN_VERSION
Definition: instanceNormalizationPlugin.cpp:64
def build_profile(builder, network, profile)
Definition: trt/util.py:310
Definition: Polygraphy/polygraphy/logger/logger.py:1
def test_simple_run(self, verbose)
Definition: tensor_quant_test.py:354
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: proposalPlugin.cpp:160
const char * getPluginType() const override
Return the plugin type.
Definition: normalizePlugin.cpp:229
def load_plugins(plugins)
Definition: trt/util.py:28
static void deserialize(void const **buffer, size_t *buffer_size, const char **value)
Definition: serialize.hpp:77
virtual ~TFusedMultiHeadAttentionXMMAKernel()=default
_learn_amax
Definition: tensor_quantizer.py:78
def mark_outputs(model, outputs)
Definition: onnx/util.py:68
virtual void destroy() noexcept=0
Destroy this object.
static half scalbln(half arg, long exp)
Scaling implementation.
Definition: samples/common/half.h:2188
int C1
Definition: nmsPlugin.h:90
upsample
Definition: demo/Tacotron2/waveglow/model.py:161
def fold_constants(self)
Definition: graph.py:341
Dims3()
Construct an empty Dims3 object.
Definition: NvInfer.h:170
float getTolerance() const
Definition: sampleConfig.h:296
Definition: optim_helper_test.py:33
int mPooledSize
Definition: multilevelCropAndResizePlugin.h:124
postnet
Definition: demo/Tacotron2/tacotron2/model.py:609
static const int32_t MAX_DIMS
The maximum number of dimensions supported for a tensor.
Definition: NvInferRuntimeCommon.h:197
virtual ~IParserError()
Definition: NvOnnxParser.h:112
@ TF_CENTER
Use box centers and size but flip x and y coordinates.
PluginFieldType type
Plugin field attribute type.
Definition: NvInferRuntimeCommon.h:861
virtual ~IIdentityLayer()
Definition: NvInfer.h:4260
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: pyramidROIAlignPlugin.cpp:292
def lookup(self, word)
Definition: cmudict.py:51
virtual size_t getWorkspaceSize(int maxBatchSize) const override
Definition: samples/opensource/samplePlugin/fcPlugin.h:164
void constructNetwork(SampleUniquePtr< nvinfer1::IBuilder > &builder, SampleUniquePtr< nvcaffeparser1::ICaffeParser > &parser, SampleUniquePtr< nvinfer1::INetworkDefinition > &network)
Uses a Caffe parser to create the MNIST Network and marks the output layers.
Definition: samplePlugin.cpp:208
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: nvFasterRCNNPlugin.cpp:244
~LSTMEncoder() override=default
def __init__(self, squad_json, vocab_file, cache_file, batch_size, max_seq_length, num_inputs)
Definition: demo/BERT/helpers/calibrator.py:28
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: batchTilePlugin.cpp:127
weight_hh
Definition: modules/quant_rnn.py:347
def inverse(self, magnitude, phase)
Definition: stft.py:107
input_size
Definition: modules/quant_rnn.py:428
def test_fine_grain(self)
Definition: calibrator_test.py:49
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleUffSSD.cpp:232
static void reportTestStart(TestAtom &testAtom)
Report that a test has started.
Definition: pluginLogging.h:345
Definition: compare.py:24
Definition: backend/pyt/runner.py:21
std::default_random_engine generator
Definition: caffeWeightFactory.h:64
virtual bool isDebug() const
Definition: parserOnnxConfig.h:137
quant_map
Definition: quant_modules.py:63
constexpr int32_t EnumMax< ElementWiseOperation >()
Maximum number of elements in ElementWiseOperation enum.
Definition: NvInfer.h:2625
virtual void setPluginFactory(IPluginFactory *factory)=0
Set the IPluginFactory used to create the user defined plugins.
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: multilevelCropAndResizePlugin.cpp:268
def main()
Definition: Tacotron2/inference.py:181
GenericBuffer(size_t size, nvinfer1::DataType type)
Construct a buffer with the specified allocation size in bytes.
Definition: buffers.h:69
int outputSize
The output size.
Definition: sampleMNISTAPI.cpp:50
def _expand_ordinal(m)
Definition: numbers.py:60
void serialize(void *buffer) const override
Serialize the layer.
Definition: multilevelProposeROIPlugin.cpp:241
virtual void markOutput(ITensor &tensor)=0
Mark a tensor as a network output.
def __call__(self)
Definition: backend/trt/loader.py:410
LogStreamConsumer(LogStreamConsumer &&other)
Definition: samples/common/logging.h:145
virtual int32_t getNbOutputs() const =0
Get the number of outputs from the layer.
std::string mCmdline
Definition: samples/common/logging.h:283
def __init__(self, dataset_path, audiopaths_and_text, args)
Definition: tacotron2/data_function.py:32
@ kHARD_SIGMOID
Hard sigmoid activation: max(0, min(1, alpha*x+beta))
void start()
Definition: samples/common/common.h:821
virtual ~IInt8Calibrator()
Definition: NvInfer.h:6056
virtual TensorFormats getAllowedFormats() const =0
Get a bitmask of TensorFormat values that the tensor supports.
SoftmaxLikelihood()=default
padding_mode
Definition: quant_conv.py:121
def parse_args(parser)
Definition: test_infer_trt.py:38
TestAtom(TestAtom &&)=default
Definition: specialSlicePlugin.h:99
def remove_weightnorm(model)
Definition: demo/Tacotron2/waveglow/model.py:273
AllocFunc allocFn
Definition: buffers.h:175
bool mShouldLog
Definition: samples/common/logging.h:104
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: specialSlicePlugin.cpp:182
Definition: reorgPlugin.h:29
def __init__(self, clip_value_min, clip_value_max, learn_min=False, learn_max=False)
Definition: clip.py:40
static const std::vector< std::pair< float, float > > ANCHOR_RATIOS
Definition: tlt_mrcnn_config.h:68
Definition: onnx-graphsurgeon/setup.py:1
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: flattenConcat.cpp:195
Definition: onnx_importer.py:69
Reference counted application-implemented error reporting interface for TensorRT objects.
Definition: NvInferRuntimeCommon.h:1233
std::string mModelFilename
Definition: sampleConfig.h:36
LogStreamConsumer gLogWarning
Definition: logger.cpp:25
const char * getPluginName() const override
Return the plugin name.
Definition: nvFasterRCNNPlugin.cpp:339
void padPPM(const PPM< uint8_t > &src, PPM< uint8_t > &dst, int top, int bottom, int left, int right)
Definition: sampleUffMaskRCNN.cpp:144
Buf buffer
Definition: checkMacrosPlugin.h:55
BertInference(const std::string &enginePath, const int maxBatchSize, const int seqLength, const bool enableGraph=false)
Definition: bert_infer.h:35
_model
Definition: backend/onnx/loader.py:186
CodeTypeSSD codeType
Definition: NvInferPluginUtils.h:186
def concatenate_layers(params)
Definition: chptToBin.py:105
int32_t embeddingVecSize
Definition: sampleMovieLens.cpp:64
Definition: sampleUffMaskRCNN.cpp:72
float * anchorsScalesHost
Definition: nvFasterRCNNPlugin.h:100
virtual uint64_t hashID(const KernelMeta &kernelMeta) const
Definition: fused_multihead_attention.h:198
virtual bool supportsFormatCombination(int32_t pos, const PluginTensorDesc *inOut, int32_t nbInputs, int32_t nbOutputs)=0
Return true if plugin supports the format and datatype for the input/output indexed by pos.
void deviceToHost(TrtCudaStream &stream)
Definition: sampleDevice.h:338
virtual ~IRuntime()
Definition: NvInferRuntime.h:768
def init_bn(module)
Definition: demo/Tacotron2/models.py:46
void sleep(int *ms)
Definition: sampleDevice.h:90
size_t mReferenceLength
Definition: bleuScoreWriter.h:54
tacotron2
Definition: export_tacotron2_onnx.py:107
BatchStream mStream
Definition: frcnnUtils.h:493
def convert_weigts(dimensions, data, forget_bias=1.0)
Definition: chptToBin.py:314
std::string dynamicRangeFileName
Definition: sampleINT8API.cpp:64
def test_raise(self)
Definition: quant_pooling_test.py:130
@ kFASTERRCNN
FasterRCNN fused plugin (RPN + ROI pooling).
std::chrono::time_point< Clock > mStop
Definition: samples/common/common.h:856
map
Definition: test_misc.py:37
def set_shapes_from_feed_dict(self, feed_dict)
Definition: backend/trt/runner.py:104
static expr atan2(float x, float y)
Arc tangent implementation.
Definition: samples/common/half.h:1935
virtual int getAttentionKeySize()=0
get the size of the attention keys
def test_clip_mode(self)
Definition: tensor_quantizer_test.py:105
def export_onnx(model, onnx_filename, batch_onnx, per_channel_quantization)
Definition: classification_flow.py:309
def print_stats(measurements_all)
Definition: test_infer.py:72
nvinfer1::DimsHW concatedBBoxDims
Definition: maskRCNNKernels.h:196
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: generateDetectionPlugin.cpp:160
Logger(Severity severity)
Definition: demo/BERT/infer_c/logging.h:31
virtual float combine(float rayLikelihood, float optionLikelihood) const =0
void destroy() override
Destroy the plugin object.
Definition: regionPlugin.cpp:398
void setPluginNamespace(const char *libNamespace) override
Set the namespace that this plugin object belongs to.
Definition: multilevelProposeROIPlugin.cpp:226
expr exp2(half arg)
Binary exponential.
Definition: ieee_half.h:2340
virtual ~IUnaryLayer()
Definition: NvInfer.h:3535
std::unordered_map< std::string, int > getBindings() const
Definition: sampleUtils.h:430
def str_from_network(network, mode="full")
Definition: trt/util.py:96
@ kPRIORBOX
PriorBox plugin.
Definition: demo/BERT/infer_c/common.h:167
@ kEXPLICIT_ROUND_UP
Use explicit padding, rounding output size up.
virtual ~Component()=default
Definition: test_data_loader.py:98
def collect_stats(model, data_loader, num_batches)
Definition: classification_flow.py:374
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleOnnxMNIST.cpp:75
def test_simple(self, resnet18)
Definition: optim_helper_test.py:97
static expr fmin(float x, float y)
Minimum implementation.
Definition: samples/common/half.h:2552
int initialize() override
Initialize the layer for execution.
Definition: qkvToContextInt8InterleavedPlugin.cpp:182
def free(self)
Definition: demo/BERT/helpers/calibrator.py:48
std::map< int32_t, std::vector< int32_t > > userToItemsMap
Definition: sampleMovieLensMPS.cpp:181
Definition: buffers.h:179
int stream
Definition: sampleReporting.h:91
std::string exportTimes
Definition: sampleOptions.h:177
def test_fake_quant_quant_per_channel_bias(self)
Definition: quant_conv_test.py:509
_scale_amax
Definition: tensor_quant.py:102
onnx_loader
Definition: backend/trt_legacy.py:127
int mBatchSize
Definition: BatchStream.h:146
bidirectional
Definition: modules/quant_rnn.py:53
def __init__(self, in_dim, out_dim, bias=True, w_init_gain='linear')
Definition: layers.py:24
def test_with_custom_quant_modules(self)
Definition: quant_modules_test.py:58
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: sampleUffPluginV2Ext.cpp:563
static PluginFieldCollection mFC
Definition: detectionLayerPlugin.h:126
int getAttentionKeySize() override
get the size of the attention keys
Definition: multiplicativeAlignment.cpp:71
bool builtin_isnan(T arg)
Check for NaN.
Definition: ieee_half.h:420
weight_hh
Definition: modules/quant_rnn.py:432
def __init__(self, n_mel_channels, n_frames_per_step, encoder_embedding_dim, attention_dim, attention_location_n_filters, attention_location_kernel_size, attention_rnn_dim, decoder_rnn_dim, prenet_dim, max_decoder_steps, gate_threshold, p_attention_dropout, p_decoder_dropout, early_stopping)
Definition: demo/Tacotron2/tacotron2/model.py:240
float class_id
Definition: sampleUffMaskRCNN.cpp:54
def parse_args(parser)
Definition: export_waveglow_onnx.py:26
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: cropAndResizePlugin.cpp:263
xy_t mFeatureSpatialSize[mFeatureMapCount]
Definition: multilevelCropAndResizePlugin.h:101
def _compute_softmax(scores)
Definition: data_processing.py:310
int32_t selectAlgorithms(const nvinfer1::IAlgorithmContext &algoContext, const nvinfer1::IAlgorithm *const *algoChoices, int32_t nbChoices, int32_t *selection) override
Return value in [0, nbChoices] for a valid algorithm.
Definition: sampleAlgorithmSelector.cpp:128
void operator()(void *ptr) const
Definition: buffers.h:210
IRuntime * createInferRuntime(ILogger &logger)
Create an instance of an IRuntime class.
Definition: NvInferRuntime.h:1990
Definition: tensor_quant.py:30
Definition: test_comparator.py:32
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: batchedNMSPlugin.cpp:459
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: proposalLayerPlugin.cpp:349
int height
Definition: batchPrepare.py:34
int inputHeight
Definition: frcnnUtils.h:58
def main()
Definition: export_waveglow_onnx.py:197
constexpr int defaultWarmUp
Definition: sampleOptions.h:47
Definition: checkMacrosPlugin.h:110
nvinfer1::DimsCHW getCHWWithExpansion(const nvinfer1::Dims &d, int filler)
Definition: parserUtils.h:114
def critical(self, message)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:244
void serialize(void *buffer) const override
Serialize the layer.
Definition: nmsPlugin.cpp:131
int enqueue(const nvinfer1::PluginTensorDesc *inputDesc, const nvinfer1::PluginTensorDesc *outputDesc, const void *const *inputs, void *const *outputs, void *workspace, cudaStream_t stream) override
Execute the layer.
Definition: embLayerNormVarSeqlenPlugin.cpp:260
def __init__(self, int num_features, float eps=1e-5, float momentum=0.1, bool affine=False, bool track_running_stats=False, **kwargs)
Definition: quant_instancenorm.py:52
static const Dims4 MODEL_MASK_SHAPE
Definition: tlt_mrcnn_config.h:175
int status
Definition: checkMacrosPlugin.h:96
int inputWidth
Definition: frcnnUtils.h:59
std::unique_ptr< Profiler > profiler
Definition: sampleInference.h:36
tokenizer
Definition: demo/BERT/helpers/calibrator.py:41
def window_sumsquare(window, n_frames, hop_length=200, win_length=800, n_fft=800, dtype=np.float32, norm=None)
Definition: audio_processing.py:23
int backtrackId
Definition: beamSearchPolicy.h:59
@ kRUNNING
The test is running.
def __init__(self, severity=INFO, colors=True, letter=True, timestamp=False, line_info=False)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:89
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: coordConvACPlugin.cpp:195
virtual ~ILayer()
Definition: NvInfer.h:945
std::vector< std::string > dataDirs
Definition: frcnnUtils.h:57
std::shared_ptr< CudaBind< int > > mValidCnt
Definition: proposalLayerPlugin.h:104
int oH
Definition: coordConvACPlugin.h:87
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: batchTilePlugin.cpp:166
void writePPMFile(const std::string &filename, PPM< uint8_t > &ppm)
Definition: sampleUffMaskRCNN.cpp:92
std::ostream & operator<<(std::ostream &o, const nvinfer1::Dims &dims)
Definition: parserUtils.h:87
LReLU PReLU
Definition: lReluPlugin.h:99
detail::enable< half &, T >::type operator*=(T rhs)
Arithmetic assignment.
Definition: ieee_half.h:1139
int64_t mUseDLACore
Definition: sampleConfig.h:50
int mKeepTopK
Definition: proposalLayerPlugin.h:99
SliceMode
Controls how ISliceLayer handles out of bounds coordinates.
Definition: NvInfer.h:3881
Definition: bboxUtils.h:26
void prepare(const int batchSize)
Definition: infer_c.cpp:31
std::string mNamespace
Definition: qkvToContextInt8InterleavedPlugin.h:79
int bbox_idx
Definition: bboxUtils.h:44
def forward(ctx, inputs, min_range, max_range, num_bits=8)
Definition: tensor_quant.py:378
@ kSAME_LOWER
Use SAME padding, with prePadding >= postPadding.
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: reorgPlugin.cpp:98
@ kUPDATE
Update gate (z).
nvinfer1::IPlugin * createPlugin(const char *layerName, const void *serialData, size_t serialLength) override
Create a plugin from serialized data.
Definition: samples/opensource/samplePlugin/fcPlugin.h:356
void printOutputParams(OutputParams &outParams)
Prints the expected recommendation results (ground truth) from the MovieLens dataset for a given user...
Definition: sampleMovieLens.cpp:458
def combine_dicts(dict0, dict1)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:53
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: pyramidROIAlignPlugin.cpp:139
int8
Definition: backend/tf/loader.py:228
SampleUniquePtr< nvinfer1::IExecutionContext > mPreprocessorContext
Definition: sampleDynamicReshape.cpp:86
bool_type< false > false_type
Definition: ieee_half.h:302
Dims mInputDims
Definition: sampleUffPluginV2Ext.cpp:638
def infer_impl(self, feed_dict)
Definition: backend/tf/runner.py:78
const char * ErrorDesc
A typedef of a c-style string for reporting error descriptions.
Definition: NvInferRuntimeCommon.h:1239
def res
Definition: classification_flow.py:443
int64_t tactic
Definition: sampleAlgorithmSelector.cpp:238
@ kCHANNEL
Elements correspond to different channels.
GenericBuffer(GenericBuffer &&buf)
Definition: buffers.h:80
int mOutputChannelCount
Definition: slpProjection.h:54
virtual int32_t getNbOutputs() const =0
Return number of outputs of the algorithm.
std::shared_ptr< CudaBind< int > > mValidCnt
Definition: detectionLayerPlugin.h:101
virtual ~SequenceProperties()=default
int mChannelVolume
Definition: groupNormalizationPlugin.h:100
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: normalizePlugin.cpp:102
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: proposalPlugin.cpp:270
def _get_network_outputs(network)
Definition: trt/util.py:186
static const std::float_round_style round_style
Rounding mode to use.
Definition: ieee_half.h:1190
def insert_suffix(path, suffix)
File creation.
Definition: Polygraphy/polygraphy/util/misc.py:282
void setPluginNamespace(const char *) override
Set the namespace that this plugin object belongs to.
Definition: split.h:119
int enqueue(int batch_size, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: multilevelCropAndResizePlugin.cpp:196
bool supportsFormatCombination(int pos, const nvinfer1::PluginTensorDesc *inOut, int nbInputs, int nbOutputs) override
doc_string
Definition: graph.py:121
virtual ~IFillLayer()
Definition: NvInfer.h:4885
DataType type
Definition: NvInferRuntimeCommon.h:370
float scoreThreshold
Definition: NvInferPluginUtils.h:251
@ kINVALID_ARGUMENT
An argument passed to the function is invalid in isolation.
def test_set_default_quant_desc(self)
Definition: quant_linear_test.py:211
Definition: tensor_quant_test.py:135
def test_fake_quant_per_channel_other_prec(self)
Definition: quant_conv_test.py:383
Definition: trt/test_loader.py:89
bert::WeightsWithOwnership mBeta
Definition: embLayerNormVarSeqlenPlugin.h:97
Definition: test_compare.py:22
nvinfer1::Dims mInputDims
The dimensions of the input to the network.
Definition: sampleUffFasterRCNN.cpp:123
static expr remainder(float x, float y)
Remainder implementation.
Definition: samples/common/half.h:1657
std::string getInfo() override
get the textual description of the component
Definition: textReader.cpp:65
def identity_identity_network()
Definition: trt/test_loader.py:53
The SampleMNISTAPIParams structure groups the additional parameters required by the SampleMNISTAPI sa...
Definition: sampleMNISTAPI.cpp:46
unsigned char fused_multihead_attention_fp16_128_64_kernel_sm80_cu_o[]
Definition: fused_multihead_attention_fp16_128_64_kernel.sm80.cpp:20
Definition: quant_pooling.py:129
infer
Definition: export_tacotron2_onnx.py:68
def _get_pronunciation(s)
Definition: cmudict.py:75
Managed buffer for host and device.
Definition: sampleDevice.h:214
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: instanceNormalizationPlugin.cpp:143
static std::vector< PluginField > mPluginAttributes
Definition: batchedNMSPlugin.h:134
def test_generator_as_data_loader(self)
Definition: test_comparator.py:52
def __init__(self, deploy, model, outputs, batch_size=None, dtype=None)
Definition: backend/trt_legacy.py:144
def setup_method(self, field_names)
Definition: test_ir.py:222
Definition: Polygraphy/tests/test_examples.py:54
virtual nvinfer1::DataType getModelDtype() const
Get the Model Data Type.
Definition: parserOnnxConfig.h:82
def indent_block(block, level=1)
Definition: Polygraphy/polygraphy/util/misc.py:416
colors
Definition: Polygraphy/polygraphy/logger/logger.py:130
size_t size() const
Returns the size (in number of elements) of the buffer.
Definition: buffers.h:128
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: cropAndResizePlugin.cpp:108
nvinfer1::Dims mInputDims
The dimensions of the input to the network.
Definition: sampleFasterRCNN.cpp:85
def __init__(self, logger, indent)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:27
static const int FPN_CLASSIF_FC_LAYERS_SIZE
Definition: tlt_mrcnn_config.h:50
virtual IPluginV2 * deserializePlugin(const char *name, const void *serialData, size_t serialLength)=0
Called during deserialization of plugin layer.
def tensors(self, check_duplicates=False)
Definition: graph.py:301
nvinfer1::DimsNCHW getDimensions() override
Definition: binaryProtoBlob.h:36
void skip(int skipCount) override
Definition: BatchStream.h:68
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:234
Dims2(int32_t d0, int32_t d1)
Construct a Dims2 from 2 elements.
Definition: NvInfer.h:83
void exportJSONTrace(const std::vector< InferenceTrace > &trace, const std::string &fileName)
Printed format: [ value, ...] value ::= { "start enq : time, "end enq" : time, "start in" : time,...
Definition: sampleReporting.cpp:221
std::vector< float > classifierRegressorStd
Definition: sampleUffFasterRCNN.cpp:75
def __call__(self)
Definition: backend/common/loader.py:30
Count_t ngramCountIntersection(const Count_t &cnt0, const Count_t &cnt1)
Definition: bleuScoreWriter.cpp:95
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: proposalLayerPlugin.cpp:187
Definition: backend/common/loader.py:19
__host__ void get_mul_shift(unsigned &mul, unsigned &shift)
Definition: reducedMathPlugin.h:89
bool processInput(const samplesCommon::BufferManager &buffers)
Reads the input and mean data, preprocesses, and stores the result in a managed buffer.
Definition: sampleUffSSD.cpp:286
Dims mOutputDims
Definition: resizeNearestPlugin.h:92
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: priorBoxPlugin.cpp:383
Definition: beamSearchPolicy.h:56
@ kFAILED_ALLOCATION
An error occurred when performing an allocation of memory on the host or the device.
int32_t selectAlgorithms(const nvinfer1::IAlgorithmContext &algoContext, const nvinfer1::IAlgorithm *const *algoChoices, int32_t nbChoices, int32_t *selection) override
Return value in [0, nbChoices] for a valid algorithm.
Definition: sampleAlgorithmSelector.cpp:283
@ kSOFTPLUS
Parametric softplus activation: alpha*log(exp(beta*x)+1)
def test_save_engine(self, identity_network)
Definition: trt/test_loader.py:271
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: pyramidROIAlignPlugin.cpp:260
int initialize() override
Initialize the layer for execution.
Definition: coordConvACPlugin.cpp:64
The UffSampleParams structure groups the additional parameters required by networks that use Uff.
Definition: argsParser.h:71
std::vector< MaskRCNNUtils::PPM< uint8_t > > mOriginalPPMs
Definition: sampleUffMaskRCNN.cpp:335
void wait()
Definition: sampleMovieLensMPS.cpp:94
_narrow_range
Definition: tensor_quantizer.py:80
bool hasSoftmaxTree
Definition: regionPlugin.h:98
LSTMDecoder(ComponentWeights::ptr weights)
Definition: lstmDecoder.cpp:29
void setEngineFileName(const char *engineFilename)
Definition: sampleConfig.h:133
void setTF32(bool enabled)
Definition: sampleConfig.h:99
virtual const char * getName() const =0
Get the tensor name.
def __len__(self)
Definition: tacotron2/data_function.py:79
def gemm(self, a, b, trans_a=False, trans_b=False)
Definition: 07_creating_a_model_with_the_layer_api/generate.py:51
def lstmcell2lstm_params(lstm_mod, lstmcell_mod)
Definition: export_tacotron2_onnx.py:85
int32_t nbInputs
Definition: sampleAlgorithmSelector.cpp:239
const char * getPluginType() const override
Return the plugin type.
Definition: embLayerNormVarSeqlenPlugin.cpp:337
int initialize() override
Initialize the layer for execution.
Definition: reorgPlugin.cpp:93
enable< bool, T, U >::type operator<(T x, U y)
Comparison for less than.
Definition: ieee_half.h:2141
def test_fake_quant_per_channel_other_precs(self)
Definition: quant_linear_test.py:153
TimePoint getEnqueueTime(bool start)
Definition: sampleInference.cpp:408
def test_keys(self, run_results)
Definition: comparator/test_struct.py:36
nvinfer1::Dims mInputDims
Definition: sampleUffMaskRCNN.cpp:332
def test_serialized_engine_loader_from_lambda(self, identity_engine)
Definition: trt/test_loader.py:73
constexpr size_t packedMaskSize384
Definition: embLayerNormVarSeqlenPlugin.cpp:43
float mScoreThreshold
Definition: generateDetectionPlugin.h:133
IPluginV2 * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: lReluPlugin.cpp:165
void setModelFileName(const char *onnxFilename)
Set the Model File Name.
Definition: sampleConfig.h:109
def test_no_quant_input_hidden_bias(self, verbose)
Definition: quant_rnn_test.py:109
int mImageSize
Definition: BatchStream.h:378
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: proposalLayerPlugin.cpp:125
def __enter__(self)
Definition: Polygraphy/polygraphy/logger/logger.py:64
virtual ~IReduceLayer()
Definition: NvInfer.h:3625
sampling_rate
Definition: tacotron2/data_function.py:36
__host__ __device__ __forceinline__ void divmod(int x, int &q, int &mod) const
Definition: reducedMathPlugin.h:80
LimitedSamplesDataReader(int maxSamplesToRead, DataReader::ptr originalDataReader)
Definition: limitedSamplesDataReader.cpp:24
int mMaxBatchSize
Definition: generateDetectionPlugin.h:100
nvinfer1::CUDADriverWrapper mDriver
Definition: fused_multihead_attention.h:275
std::shared_ptr< softmaxTree > smTree
Definition: regionPlugin.h:96
nvinfer1::IPluginV2 * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:331
def forward(self, input, hx=None)
Definition: modules/quant_rnn.py:407
float * minSize
Definition: NvInferPluginUtils.h:93
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/exception.py:17
Definition: tools/pytorch-quantization/pytorch_quantization/calib/calibrator.py:21
Definition: pyramidROIAlignPlugin.h:104
int32_t calBatchSize
The size of calibration batch.
Definition: sampleUffSSD.cpp:53
std::unordered_map< std::string, ShapeRange > shapesCalib
Definition: sampleOptions.h:124
def test_quant_input_hidden_bias(self, verbose)
Definition: quant_rnn_test.py:448
const PluginField * fields
Pointer to PluginField entries.
Definition: NvInferRuntimeCommon.h:879
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: priorBoxPlugin.cpp:308
virtual void reportLayerTime(const char *layerName, float ms)=0
Layer time reporting callback.
std::string getInfo() override
get the textual description of the component
Definition: beamSearchPolicy.cpp:166
void addToModel(nvinfer1::INetworkDefinition *network, nvinfer1::ITensor *inputFromDecoder, nvinfer1::ITensor *context, nvinfer1::ITensor **attentionOutput) override
add the attention vector calculation to the network
Definition: slpAttention.cpp:38
def resize(self, shape)
Definition: cuda.py:146
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: cropAndResizePlugin.cpp:200
def percentage(self, runner_pair=None)
Definition: comparator/struct.py:173
Definition: profiler.py:1
int mV
Definition: sampleUffPluginV2Ext.cpp:181
void destroy() override
Destroy the plugin object.
Definition: embLayerNormVarSeqlenPlugin.cpp:395
BufferDesc(nvinfer1::Dims dims, int dataWidth, TensorFormat format)
Definition: sampleReformatFreeIO.cpp:69
virtual bool setDynamicRange(const char *tensorName, float min, float max)=0
Update dynamic range for a tensor.
unsigned int mMetaInfoIndex
Definition: fused_multihead_attention.h:284
bias
Definition: modules/quant_rnn.py:49
~FlattenConcat() override
Definition: flattenConcat.cpp:80
def test_repr(self)
Definition: calibrator_test.py:90
Definition: onnx_importer.py:1
bool getPrintLayerInfo() const
Get whether the layer information will be printed.
Definition: sampleConfig.h:161
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: normalizePlugin.cpp:87
static const char * SKIP_LAYER_NORM_INTERLEAVED_VERSION
Definition: skipLayerNormInt8InterleavedPlugin.cpp:41
bool teardown()
Used to clean up any state created in the sample class.
Definition: sampleUffPluginV2Ext.cpp:326
Definition: quant_linear.py:28
nvinfer1::DimsHW preRefineSortedScoreDims
Definition: maskRCNNKernels.h:162
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: groupNormalizationPlugin.cpp:246
int32_t anchorsRatioCount
Definition: NvInferPluginUtils.h:123
path
Definition: backend/trt_legacy.py:37
float y2
Definition: samples/common/common.h:669
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: proposalPlugin.cpp:248
int keepTopK
The maximum number of detection post-NMS.
Definition: sampleSSD.cpp:50
LogStreamConsumerBase(std::ostream &stream, const std::string &prefix, bool shouldLog)
Definition: samples/common/logging.h:114
const char * NMS_PLUGIN_VERSION
Definition: batchedNMSPlugin.cpp:34
int32_t numMaxSize
Definition: NvInferPluginUtils.h:94
virtual nvinfer1::ICudaEngine * deserializeCudaEngine(const void *blob, std::size_t size, IPluginFactory *pluginFactory) noexcept=0
Deserialize an engine from a stream.
const void * readCalibrationCache(size_t &length) override
Definition: sampleEngines.cpp:239
def test_fake_quant_against_unquantized(self)
Definition: quant_linear_test.py:178
def mergeHeaders(features, skipFirst=True)
Definition: profiler.py:99
float out
Definition: sampleReporting.h:55
static const char * PROPOSAL_PLUGIN_NAME
Definition: proposalPlugin.cpp:33
Descriptor for two-dimensional data.
Definition: NvInfer.h:65
CUresult cuLinkAddFile(CUlinkState state, CUjitInputType type, const char *path, unsigned int numOptions, CUjit_option *options, void **optionValues) const
Definition: cudaDriverWrapper.cpp:112
Definition: backend/trt/loader.py:1
@ kSKIP
No operation is performed on the first recurrent layer.
output_node
Definition: test_ir.py:125
def __init__(self, attention_n_filters, attention_kernel_size, attention_dim)
Definition: demo/Tacotron2/tacotron2/model.py:30
constexpr size_t packedMaskSize128
Definition: embLayerNormVarSeqlenPlugin.cpp:41
size_t getSize() const
Definition: sampleDevice.h:343
size_t totalSize
Definition: maskRCNNKernels.h:205
virtual int32_t getConstantValue() const =0
If isConstant(), returns value of the constant.
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: resizeNearestPlugin.cpp:62
def make_iter_results(runner_name)
Definition: comparator/test_struct.py:9
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: groupNormalizationPlugin.cpp:77
Definition: groupNormalizationPlugin.h:119
static double lgamma(double arg)
Definition: samples/common/half.h:2485
std::vector< std::unique_ptr< Iteration > > IterationStreams
Definition: sampleInference.cpp:474
~CudaBind()
Definition: maskRCNNKernels.h:74
def test_input_fake_quant_disable(self)
Definition: quant_pooling_test.py:186
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: normalizePlugin.cpp:117
Definition: sampleUffPluginV2Ext.cpp:337
int hiddenSize
Definition: sampleCharRNN.cpp:117
Definition: comparator/data_loader.py:25
Definition: tools/Polygraphy/polygraphy/backend/common/__init__.py:1
bool skip
Definition: sampleOptions.h:159
half rint(half arg)
Nearest integer using half's internal rounding mode.
Definition: ieee_half.h:2593
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: coordConvACPlugin.cpp:166
W_inverse
Definition: demo/Tacotron2/waveglow/model.py:77
def skipTrace(trace, start)
Definition: tracer.py:53
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: normalizePlugin.cpp:184
void check_valid_inputs(const nvinfer1::Dims *inputs, int nbInputDims)
Definition: detectionLayerPlugin.cpp:217
Definition: quant_bert.py:49
Definition: demo/Tacotron2/tacotron2/model.py:108
def main()
Definition: infer.py:43
@ kCAFFE_ROUND_DOWN
Use CAFFE padding, rounding output size down, uses prePadding value.
std::vector< nvinfer1::Dims > getStateSizes() override
get the sizes (vector of them) of the hidden state vectors
Definition: lstmEncoder.cpp:105
def indent(self, level=1)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:135
def read_histogram_cache(self, length)
Definition: demo/BERT/helpers/calibrator.py:107
RegionPluginCreator()
Definition: regionPlugin.cpp:468
const char * getPluginVersion() const override
Return the plugin version.
Definition: priorBoxPlugin.cpp:373
Definition: onnx/test_loader.py:53
@ kCDHW32
Thirty-two wide channel vectorized row major format.
~RPROIPluginCreator() override
Definition: nvFasterRCNNPlugin.cpp:334
uff_path
Definition: backend/trt_legacy.py:51
bool verbose
Definition: sampleINT8API.cpp:57
virtual ~IProfiler()
Definition: NvInferRuntime.h:676
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: normalizePlugin.cpp:139
CudaBind(size_t size)
Definition: maskRCNNKernels.h:68
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/exception.py:1
int initialize() override
Initialize the layer for execution.
Definition: gridAnchorPlugin.cpp:186
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: resizeNearestPlugin.cpp:119
_skip_zeros
Definition: histogram.py:47
@ kCHW16
Sixteen wide channel vectorized row major format.
~SoftmaxLikelihood() override=default
virtual ~UffPoolPluginV2()
Definition: sampleUffPluginV2Ext.cpp:383
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: regionPlugin.cpp:438
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: flattenConcat.cpp:284
def test_simple(self)
Definition: optim_helper_test.py:105
def __str__(self)
Definition: max.py:93
uint16 int2half(T value)
Convert integer to half-precision floating point.
Definition: ieee_half.h:737
def main()
Definition: export_onnx2trt.py:48
def test_device_buffer_memcpy_async(self)
Definition: test_cuda.py:50
def extend(self, Sequence[object] iterable)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:102
Dims max
Upper bounds on tensor’s dimensions.
Definition: NvInferRuntime.h:396
BatchedNMSBasePluginCreator()
Definition: batchedNMSPlugin.cpp:469
Definition: backend/pyt/runner.py:1
virtual int getOutputSize()=0
get the size of raw logits vector
virtual void terminate()=0
Release resources acquired during plugin layer initialization.
samplesCommon::CaffeSampleParams mParams
The parameters for the sample.
Definition: samplePlugin.cpp:109
virtual float getDynamicRangeMax() const =0
Get maximum of dynamic range.
virtual void destroy()=0
destroy this object
static std::ostream & severityOstream(Severity severity)
Definition: samples/common/logging.h:160
std::vector< float > mLabels
Definition: BatchStream.h:151
decoder
Definition: demo/Tacotron2/tacotron2/model.py:600
Definition: checkMacrosPlugin.h:49
IPluginV2DynamicExt * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: batchedNMSPlugin.cpp:569
engine_dir
Definition: backend/tf/loader.py:325
@ kCHW32
Thirty-two wide channel vectorized row major format.
nvinfer1::Weights mKernelWeights
Definition: slpAttention.h:48
@ kSSDDETECTIONOUTPUT
SSD DetectionOutput plugin.
def test_basic_forward(self, verbose)
Definition: quant_rnn_test.py:266
def __enter__(self)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:32
def name(self)
Definition: tensor_quant.py:136
nvinfer1::DimsExprs getOutputDimensions(int outputIndex, const nvinfer1::DimsExprs *inputs, int nbInputs, nvinfer1::IExprBuilder &exprBuilder) override
Definition: qkvToContextInt8InterleavedPlugin.cpp:102
Definition: sampleEngines.cpp:51
const char * CELL_OUT_BLOB_NAME
Definition: sampleCharRNN.cpp:91
void * packed_mask_ptr
Definition: fused_multihead_attention.h:53
virtual void addToModel(nvinfer1::INetworkDefinition *network, nvinfer1::ITensor *inputData, nvinfer1::ITensor **inputStates, nvinfer1::ITensor **outputData, nvinfer1::ITensor **outputStates)=0
add the memory, cell, and hidden states to the network
float combine(float rayLikelihood, float optionLikelihood) const override
Definition: softmaxLikelihood.cpp:82
void recordEnqueueTime()
Definition: sampleInference.cpp:402
location_conv
Definition: demo/Tacotron2/tacotron2/model.py:33
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: specialSlicePlugin.cpp:72
InferenceTime(float q, float i, float c, float o, float e)
Definition: sampleReporting.h:36
CudaError(const char *fl, const char *fn, int ln, int stat, const char *msg=nullptr)
Definition: checkMacrosPlugin.h:104
std::vector< T > buffer
Definition: sampleUffMaskRCNN.cpp:76
def test_cuda_ext(self)
Definition: tensor_quant_test.py:198
SpecialSlice()
Definition: specialSlicePlugin.cpp:123
opset
Definition: onnx_models.py:40
nvinfer1::Dims mOutputDims
The dimensions of the output to the network.
Definition: sampleOnnxMNIST.cpp:72
DimensionOperation
An operation on two IDimensionExpr, which represent integer expressions used in dimension computation...
Definition: NvInferRuntime.h:295
constexpr int defaultMinTiming
Definition: sampleOptions.h:37
static void addDumpTensorToStream(nvinfer1::INetworkDefinition *network, nvinfer1::ITensor *input, nvinfer1::ITensor **output, std::shared_ptr< std::ostream > out)
Definition: debugUtil.cpp:123
orginal_func_map
Definition: quant_modules.py:56
void serialize(void *buffer) const override
Serialize the layer.
Definition: gridAnchorPlugin.cpp:226
int dtype_to_onnx(np.dtype dtype)
Definition: onnx_exporter.py:27
const char * getPluginVersion() const override
Return the plugin version.
Definition: reorgPlugin.cpp:110
~RegionPluginCreator() override=default
const char * getPluginVersion() const override
Return the plugin version.
Definition: flattenConcat.cpp:306
const char * NORMALIZE_PLUGIN_NAME
Definition: normalizePlugin.cpp:32
weights storage
Definition: componentWeights.h:31
virtual const IAlgorithmIOInfo & getAlgorithmIOInfo(int32_t index) const =0
Returns the format of an Algorithm input or output.
IPluginV2Ext * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: batchTilePlugin.cpp:211
Definition: test_ir.py:450
void destroy() override
Destroy the plugin object.
Definition: priorBoxPlugin.cpp:244
bool mRNNKind
Definition: lstmEncoder.h:53
@ kMIN
Minimum of the two elements.
cudnnHandle_t _cudnn_handle
Definition: instanceNormalizationPlugin.h:97
IRefitter * createInferRefitter(ICudaEngine &engine, ILogger &logger)
Create an instance of an IRefitter class.
Definition: NvInferRuntime.h:2000
virtual void setModelFileName(const char *onnxFilename)
Set the Model File Name.
Definition: parserOnnxConfig.h:91
static half negate(half arg)
Negation implementation.
Definition: samples/common/half.h:2516
Dims min
Lower bounds on tensor’s dimensions.
Definition: NvInferRuntime.h:393
bool setUpInference(InferenceEnvironment &iEnv, const InferenceOptions &inference)
Set up contexts and bindings for inference.
Definition: sampleInference.cpp:41
float nmsIouThresholdClassifier
Definition: sampleUffFasterRCNN.cpp:73
def __init__(self, vocab, unk_token="[UNK]", max_input_chars_per_word=200)
Definition: tokenization.py:333
virtual ~IPluginV2Ext()
Definition: NvInferRuntimeCommon.h:654
int outputClsSize
The number of output classes.
Definition: sampleFasterRCNN.cpp:47
virtual ErrorCode code() const =0
the error code
def forward(self, decoder_input, attention_hidden, attention_cell, decoder_hidden, decoder_cell, attention_weights, attention_weights_cum, attention_context, memory, processed_memory, mask)
Definition: export_tacotron2_onnx.py:181
@ kMIN
This is used to set or get the minimum permitted value for dynamic dimensions etc.
def scale(self)
Definition: tensor_quantizer.py:118
def amax(self)
Definition: tensor_quantizer.py:126
int mBatch
Definition: sampleInference.cpp:207
value
Definition: test_func.py:121
def __init__(self, kernel_size, stride=None, padding=0, dilation=1, return_indices=False, ceil_mode=False, **kwargs)
Definition: quant_pooling.py:51
def from_yaml(cls, yaml_str)
Definition: tensor_quant.py:211
def test_fake_quant_per_channel_bias(self)
Definition: quant_conv_transposed_test.py:190
attention_rnn
Definition: export_tacotron2_onnx.py:116
bool checkEraseNegativeOption(Arguments &arguments, const std::string &option, bool &value)
Definition: sampleOptions.cpp:192
def __call__(self)
Definition: backend/trt/loader.py:505
def test_values(self, run_results)
Definition: comparator/test_struct.py:40
int dlaCore
Definition: sampleMovieLens.cpp:526
virtual ~IPluginExt()
Definition: NvInferRuntime.h:273
int mBatches
Definition: sampleEngines.cpp:183
def to_gpu(x)
Definition: demo/Tacotron2/common/utils.py:50
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: lReluPlugin.cpp:105
std::shared_ptr< trtcaffe::NetParameter > mDeploy
Definition: caffeParser.h:82
def test_extend_named_parameters(self)
Definition: test_func.py:33
outputs
Definition: backend/trt_legacy.py:39
def toposort_multi_tier_output_graph()
Definition: test_ir.py:405
const char * getPluginName() const override
Return the plugin name.
Definition: normalizePlugin.cpp:267
BufferManager(std::shared_ptr< nvinfer1::ICudaEngine > engine, const int batchSize=0, const nvinfer1::IExecutionContext *context=nullptr)
Create a BufferManager for handling buffer interactions with engine.
Definition: buffers.h:245
std::vector< MaskRCNNUtils::PPM< uint8_t > > mPPMs
Definition: sampleUffMaskRCNN.cpp:338
def __init__(self, graph, config=None)
Definition: backend/tf/loader.py:393
_num_bins
Definition: histogram.py:46
Definition: onnx_exporter.py:30
bool bnConvertWrap(float scaleFactor, const Weights &variance, const Weights &mean, const Weights &scaleBlob, const Weights &biasBlob, Weights &shift, Weights &scale, float eps, bool nvCaffe, CaffeWeightFactory &weightFactory)
Definition: parseBatchNorm.cpp:24
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:26
def test_unsigned_one_tensor(self, verbose)
Definition: calibrator_test.py:265
def __enter__(self)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:47
The SampleCharRNNBase class implements the char_rnn sample.
Definition: sampleCharRNN.cpp:142
int layerCount
Definition: sampleCharRNN.cpp:116
size_t getWorkspaceSize(int) const override
Definition: pyramidROIAlignPlugin.cpp:107
def __str__(self)
Definition: onnx_models.py:88
void finalize() override
it is called right after inference ends
Definition: textWriter.cpp:38
static const char * PROPOSAL_PLUGIN_VERSION
Definition: proposalPlugin.cpp:32
bias_hh
Definition: modules/quant_rnn.py:394
float scale
Definition: NvInferRuntimeCommon.h:372
def set_output_name(layer, prefix, name, out_idx=0)
Definition: builder.py:109
void wait(TrtCudaEvent &event)
Definition: sampleDevice.h:156
Definition: quant_pooling_test.py:218
const char * getPluginType() const override
Return the plugin type.
Definition: reorgPlugin.cpp:105
size_t mRpnWidth
Definition: proposalPlugin.h:112
def test_lr_momentum_decay(self, resnet18)
Definition: optim_helper_test.py:57
void read(const char *&buffer, T &val)
Definition: samples/opensource/samplePlugin/fcPlugin.h:237
int C
Definition: reorgPlugin.h:87
int32_t getTensorRTVersion() const
Return the API version with which this plugin was built.
Definition: NvInferRuntime.h:534
virtual bool setDynamicRange(float min, float max)=0
Set dynamic range for the tensor.
float * anchorsRatiosHost
Definition: nvFasterRCNNPlugin.h:100
values
Definition: tensor.py:70
nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: embLayerNormVarSeqlenPlugin.cpp:324
static PluginFieldCollection mFC
Definition: pyramidROIAlignPlugin.h:122
int64_t qkv_stride_in_bytes
Definition: fused_multihead_attention_v2.h:35
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: batchTilePlugin.cpp:204
def test_unsigned(self)
Definition: tensor_quant_test.py:191
def is_dimension_dynamic(dim)
Definition: trt_utils.py:20
void fill(const std::string &fileName)
Definition: sampleUtils.h:222
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: normalizePlugin.cpp:81
bool mStarted
Definition: samples/common/logging.h:281
def test_cleanup_graph_input_producers(self)
Definition: test_ir.py:653
Definition: sampleINT8API.cpp:44
SampleFasterRCNN(const SampleFasterRCNNParams ¶ms)
Definition: sampleFasterRCNN.cpp:61
@ kINT32
Signed 32-bit integer format.
void setReportableSeverity(Severity reportableSeverity)
Definition: samples/common/logging.h:153
Definition: tools/pytorch-quantization/pytorch_quantization/nn/modules/__init__.py:1
nvinfer1::IPluginV2 * createPlugin(const char *name, const nvinfer1::PluginFieldCollection *fc) override
Return a plugin object.
Definition: embLayerNormVarSeqlenPlugin.cpp:440
def extra_repr(self)
Definition: tensor_quantizer.py:362
def __init__(self, str name, np.ndarray values)
Definition: tensor.py:187
bool isPluginExt(const char *name) override
A user implemented function that determines if a layer configuration is provided by an IPluginExt.
Definition: samples/opensource/samplePlugin/fcPlugin.h:330
def test_extend_0_args_2_rv(self)
Definition: test_func.py:57
int H
Definition: regionPlugin.h:97
expr asinh(half arg)
Hyperbolic area sine.
Definition: ieee_half.h:2494
def test_fake_quant_per_tensor(self)
Definition: quant_conv_transposed_test.py:412
Definition: test_func.py:19
void destroy() override
Destroy the plugin object.
Definition: proposalPlugin.cpp:250
def convert_doc_tokens(paragraph_text)
Definition: data_processing.py:26
void wait(TrtCudaEvent &gpuStart)
Definition: sampleInference.cpp:366
def reset_logger_handler()
Definition: quant_logging.py:22
def emb_layernorm(builder, network, config, weights_dict, builder_config, max_sequence_length, max_batch_size)
Definition: builder_varseqlen.py:510
def __call__(self)
Definition: backend/trt_legacy.py:87
Definition: modules/quant_rnn.py:299
seen_tensors
Definition: graph.py:32
unsigned char fused_multihead_attention_v2_fp16_64_64_kernel_sm75_cubin[]
Definition: fused_multihead_attention_v2_fp16_64_64_kernel.sm75.cpp:19
void * handle
Definition: cudaDriverWrapper.h:69
void write(const int *hOutputData, int actualOutputSequenceLength, int actualInputSequenceLength) override
write the generated sequence
Definition: benchmarkWriter.cpp:32
int outputClsSize
The number of output classes.
Definition: sampleSSD.cpp:49
def tokenize(self, text)
Definition: tokenization.py:169
void beginCapture(TrtCudaStream &stream)
Definition: sampleDevice.h:186
std::shared_ptr< Alignment > ptr
Definition: alignment.h:35
std::string mCacheFile
Definition: sampleEngines.cpp:185
bool helps
Definition: sampleOptions.h:193
Definition: mrcnn_to_trt_single.py:93
void * mBuffer
Definition: buffers.h:174
@ kCHANNEL
Per-channel coefficients.
softmaxTree
Definition: NvInferPluginUtils.h:220
unsigned int fused_multihead_attention_v2_fp16_128_64_kernel_sm75_cubin_len
Definition: fused_multihead_attention_v2_fp16_128_64_kernel.sm75.cpp:6578
num_hidden_layers
Definition: builder_varseqlen.py:96
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: specialSlicePlugin.cpp:172
int mBatchSize
Definition: BatchStream.h:373
def prepare_model(model_name, data_dir, per_channel_quantization, batch_size_train, batch_size_test, batch_size_onnx, calibrator, pretrained=True, ckpt_path=None, ckpt_url=None)
Definition: classification_flow.py:91
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: instanceNormalizationPlugin.cpp:290
def test_quant_input_hidden_bias(self, verbose)
Definition: quant_rnn_test.py:193
const char * getPluginVersion() const override
Return the plugin version.
Definition: qkvToContextInt8InterleavedPlugin.cpp:287
size_t getWorkspaceSize(int) const override
Definition: flattenConcat.cpp:128
std::shared_ptr< Encoder > ptr
Definition: encoder.h:36
static const char * testResultString(TestResult result)
returns an appropriate string for prefixing a test result message with the given result
Definition: samples/common/logging.h:395
outputs
Definition: isolate.py:55
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: regionPlugin.cpp:239
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleUffPluginV2Ext.cpp:192
def pop(self, i=-1)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:117
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: gridAnchorPlugin.cpp:316
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: resizeNearestPlugin.cpp:240
unsigned int fused_multihead_attention_v2_int8_384_64_kernel_cubin_len
Definition: fused_multihead_attention_v2_int8_384_64_kernel.sm72.cpp:19647
std::vector< nvinfer1::PluginField > parseLReLUParam(const trtcaffe::LayerParameter &msg, CaffeWeightFactory &weightFactory, BlobNameToTensor &tensors)
Definition: caffeParser.cpp:236
nvinfer1::DimsHW sortClassSampleIdxDims
Definition: maskRCNNKernels.h:134
virtual bool parseFromFile(const char *onnxModelFile, int verbosity)=0
Parse an onnx model file, can be a binary protobuf or a text onnx model calls parse method inside.
SampleCharRNNMaps charMaps
Definition: sampleCharRNN.cpp:127
@ kASINH
Inverse hyperbolic sine.
nvinfer1::Weights getNullWeights()
Definition: caffeWeightFactory.cpp:194
Definition: fused_multihead_attention_v2.h:421
std::vector< std::string > outputTensorNames
Definition: argsParser.h:44
def __setattr__(self, name, value)
Definition: tensor.py:35
const char * PYRAMIDROIALGIN_PLUGIN_NAME
Definition: pyramidROIAlignPlugin.cpp:29
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:262
virtual void setName(const char *name)=0
Set the tensor name.
void addBinding(int b, const std::string &name, bool isInput, int volume, nvinfer1::DataType dataType, const std::string &fileName="")
Definition: sampleUtils.h:307
virtual IConcatenationLayer * addConcatenation(ITensor *const *inputs, int32_t nbInputs)=0
Add a concatenation layer to the network.
def __init__(self, model_bytes)
Definition: backend/onnxrt/loader.py:23
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: nvFasterRCNNPlugin.cpp:122
virtual void destroy()=0
Destroy this ICaffeParser object.
std::shared_ptr< CudaBind< int > > mValidCnt
Definition: multilevelProposeROIPlugin.h:103
Definition: quant_utils_test.py:30
Definition: sampleAlgorithmSelector.cpp:235
int b
Definition: fused_multihead_attention_v2.h:56
float mNmsIouThreshold
Definition: proposalPlugin.h:116
def test_unsigned(self)
Definition: tensor_quant_test.py:99
int mBackgroundLabel
Definition: multilevelProposeROIPlugin.h:94
CUresult(* _cuModuleLoadData)(CUmodule *, const void *)
Definition: cudaDriverWrapper.h:76
Definition: trt/test_loader.py:270
def print_single_query(eval_time_elapsed, prediction, nbest_json)
Definition: BERT/inference.py:216
engine_path
Definition: backend/trt_legacy.py:221
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: samplePlugin.cpp:233
int initialize() override
Initialize the layer for execution.
Definition: cropAndResizePlugin.cpp:126
unsigned int fused_multihead_attention_v2_int8_192_64_kernel_cubin_len
Definition: fused_multihead_attention_v2_int8_192_64_kernel.sm72.cpp:15999
const nvinfer1::PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: qkvToContextInt8InterleavedPlugin.cpp:292
virtual void addToModel(nvinfer1::INetworkDefinition *network, int beamWidth, nvinfer1::ITensor *inputLogits, nvinfer1::ITensor *inputLikelihoods, nvinfer1::ITensor **newCombinedLikelihoods, nvinfer1::ITensor **newRayOptionIndices, nvinfer1::ITensor **newVocabularyIndices)=0
add calculation of likelihood and TopK indices to the network
def register_callback()
Definition: tools/Polygraphy/polygraphy/backend/tf/__init__.py:5
def test_calibrator_outside_polygraphy(self, identity_builder_network)
Definition: test_calibrator.py:68
def forward(self, sequence, sequence_lengths)
Definition: export_tacotron2_onnx.py:70
~SampleBuffer()
Definition: sampleReformatFreeIO.cpp:155
std::vector< int > batchSize
Definition: demo/BERT/infer_c/common.h:30
std::ostream & printTacticSources(std::ostream &os, nvinfer1::TacticSources enabledSources, nvinfer1::TacticSources disabledSources)
Definition: sampleOptions.cpp:340
void setPluginNamespace(const char *libNamespace) override
Set the namespace of the plugin creator based on the plugin library it belongs to.
Definition: batchTilePlugin.h:99
virtual Dims getBindingDimensions(int32_t bindingIndex) const noexcept=0
Get the dynamic dimensions of a binding.
static void serialize(void **buffer, const char *value)
Definition: serialize.hpp:72
void addToModel(nvinfer1::INetworkDefinition *network, nvinfer1::ITensor *attentionKeys, nvinfer1::ITensor *queryStates, nvinfer1::ITensor **alignmentScores) override
add the alignment scores calculation to the network
Definition: multiplicativeAlignment.cpp:38
int_range
Definition: comparator/data_loader.py:89
bool build()
Function builds the network engine.
Definition: sampleFasterRCNN.cpp:132
def _quant_forward(self, inputs)
Definition: tensor_quantizer.py:299
std::ostream & operator<<(std::ostream &os, const BaseModelOptions &options)
Definition: sampleOptions.cpp:836
Measurement points in milliseconds.
Definition: sampleReporting.h:69
def run_trt_engine(context, engine, tensors)
Definition: trt_utils.py:28
static std::vector< PluginField > mPluginAttributes
Definition: gridAnchorPlugin.h:124
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: flattenConcat.cpp:198
~Likelihood() override=default
std::vector< std::shared_ptr< CudaBind< float > > > mTempScores
Definition: multilevelProposeROIPlugin.h:107
bool varianceEncodedInTarget
Definition: NvInferPluginUtils.h:183
int initialize() override
Initialize the layer for execution.
Definition: debugUtil.cpp:70
void destroy() override
Destroy the plugin object.
Definition: reorgPlugin.cpp:115
def test_can_name_runner(self)
Definition: onnxrt/test_runner.py:31
Definition: functional.py:1
int mInputChannelCount
Definition: slpProjection.h:53
def test_setters(self)
Definition: tensor_quantizer_test.py:238
static const std::vector< std::string > MODEL_OUTPUTS
Definition: tlt_mrcnn_config.h:173
std::string mNameSpace
Definition: pyramidROIAlignPlugin.h:101
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: lReluPlugin.cpp:46
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: priorBoxPlugin.h:47
std::shared_ptr< trtcaffe::NetParameter > mModel
Definition: caffeParser.h:83
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleMLP.cpp:92
ErrorCode getErrorCode(int32_t errorIdx) const noexcept final
Returns the ErrorCode enumeration.
Definition: ErrorRecorder.h:50
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: detectionLayerPlugin.cpp:237
int enqueue(const nvinfer1::PluginTensorDesc *inputDesc, const nvinfer1::PluginTensorDesc *outputDesc, const void *const *inputs, void *const *outputs, void *workspace, cudaStream_t stream) override
Execute the layer.
Definition: instanceNormalizationPlugin.cpp:172
Definition: proposalLayerPlugin.h:116
Definition of plugin versions.
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: coordConvACPlugin.cpp:155
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: nmsPlugin.cpp:177
void shutdownProtobufLibrary()
Shuts down protocol buffers library.
Definition: NvCaffeParser.cpp:22
Definition: sampleMovieLensMPS.cpp:155
IParser * createParser(nvinfer1::INetworkDefinition &network, nvinfer1::ILogger &logger)
Create a new parser object.
Definition: NvOnnxParser.h:260
int32_t imgW
Definition: NvInferPluginUtils.h:98
virtual Dims getDimensions() const =0
Get the dimensions of a tensor.
constexpr size_t packedMaskSize256
Definition: embLayerNormVarSeqlenPlugin.cpp:42
stft_fn
Definition: layers.py:62
def test_print_module(self)
Definition: print_test.py:43
Definition: component.h:23
int _nz
Definition: split.h:42
std::vector< EnqueueTimes > mEnqueueTimes
Definition: sampleInference.cpp:471
def extend(extend_func)
Definition: func.py:25
FieldType
The possible field types for custom layer.
Definition: NvUffParser.h:57
bool acrossSpatial
Definition: normalizePlugin.h:98
@ kMIN
Minimum of the two operands.
ModelOptions model
Definition: sampleOptions.h:188
def test_single_key(self, resnet18)
Definition: optim_helper_test.py:35
std::vector< IOFormat > outputFormats
Definition: sampleOptions.h:126
virtual void setFullTextFileName(const char *fullTextFilename)
Set the File Name of the Network Description as a Text File, including the weights.
Definition: parserOnnxConfig.h:124
virtual const char * func() const =0
source function in which the error occurred
std::vector< int > mCuSeqlens
Definition: bert_infer.h:344
void write(const int *hOutputData, int actualOutputSequenceLength, int actualInputSequenceLength) override
write the generated sequence
Definition: textWriter.cpp:30
nvinfer1::DataType mType
Definition: multilevelProposeROIPlugin.h:114
const void * data
Plugin field attribute data.
Definition: NvInferRuntimeCommon.h:856
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: multilevelCropAndResizePlugin.cpp:294
def basic_compare_func(check_shapes=None, rtol=None, atol=None, fail_fast=None, find_output_func=None)
Definition: compare.py:64
def encoder_infer(self, x, input_lengths)
Definition: export_tacotron2_onnx.py:43
nvinfer1::IPluginV2 * createPlugin(const char *name, const nvinfer1::PluginFieldCollection *fc) override
Return a plugin object.
Definition: qkvToContextInt8InterleavedPlugin.cpp:297
bool processInput(const samplesCommon::BufferManager &buffers, const std::string &inputTensorName, int inputFileIdx) const
Reads the input and mean data, preprocesses, and stores the result in a managed buffer.
Definition: samplePlugin.cpp:294
const char * getTextFileName() const
Returns the File Name of the Network Description as a Text File.
Definition: sampleConfig.h:137
std::string mNamespace
Definition: coordConvACPlugin.h:114
Definition: sampleCharRNN.cpp:85
expr sin(half arg)
Sine function.
Definition: ieee_half.h:2417
bool broadcastIOFormats(const std::vector< IOFormat > &formats, size_t nbBindings, bool isInput=true)
Definition: sampleUtils.h:468
Definition: onnx/test_runner.py:21
virtual ~IPluginFactoryV2()
Definition: NvCaffeParser.h:158
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: nvFasterRCNNPlugin.cpp:249
SampleINT8APIParams mParams
Stores Sample Parameter.
Definition: sampleINT8API.cpp:103
int mConcatAxisID
Definition: flattenConcat.h:128
def test_raise(self)
Definition: clip_test.py:46
@ kCONSTANT
weights for IConstantLayer
const char * boolToEnabled(bool enable)
Definition: sampleOptions.cpp:171
virtual nvinfer1::ITensor * find(const char *name) const =0
Given a blob name, returns a pointer to a ITensor object.
int32_t order[Dims::MAX_DIMS]
The elements of the permutation.
Definition: NvInfer.h:3734
static half ceil(half arg)
Ceiling implementation.
Definition: samples/common/half.h:2086
processes the results of one iteration of the generator with beam search and produces input for the n...
Definition: beamSearchPolicy.h:33
Definition: quant_pooling_test.py:239
const char * getPluginNamespace() const override
Return the namespace of the plugin creator object.
Definition: groupNormalizationPlugin.cpp:275
def log_module_info(module, name=None, severity=G_LOGGER.VERBOSE)
Definition: Polygraphy/polygraphy/util/misc.py:401
void synchronize()
Definition: sampleDevice.h:139
def __init__(self)
Definition: tacotron2/loss_function.py:21
Definition: regionPlugin.h:30
def test_no_quant(self)
Definition: quant_conv_test.py:269
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:55
std::shared_ptr< std::istream > mReferenceInput
Definition: bleuScoreWriter.h:52
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: cropAndResizePlugin.cpp:194
Definition: quant_conv.py:350
const char * name
Plugin field attribute name.
Definition: NvInferRuntimeCommon.h:852
~LReLU() override=default
def __call__(self)
Definition: backend/onnxrt/loader.py:33
virtual void skip(int skipCount)=0
def num_bits(self)
Definition: tensor_quantizer.py:110
num_samples
Definition: stft.py:81
Definition: constants.py:1
std::vector< std::string > mDataDir
Directories where the files can be found.
Definition: BatchStream.h:387
T roundUp(T m, T n)
Definition: sampleUtils.h:59
def test_deep_copy(self)
Definition: test_ir.py:682
def main()
Definition: builder.py:646
Definition: embLayerNormVarSeqlenPlugin.h:115
void batch_inverse_transform_classifier(const float *roi_after_nms, int roi_num_per_img, const float *classifier_cls, const float *classifier_regr, std::vector< float > &pred_boxes, std::vector< int > &pred_cls_ids, std::vector< float > &pred_probs, std::vector< int > &box_num_per_img, int N)
Helper function to do post-processing(apply delta to ROIs).
Definition: sampleUffFasterRCNN.cpp:522
def set_default_quant_desc_weight(cls, value)
Definition: _utils.py:47
CUresult(* _cuLinkComplete)(CUlinkState, void **, size_t *)
Definition: cudaDriverWrapper.h:72
Definition: calibrator_test.py:249
virtual void configurePlugin(const Dims *inputDims, int32_t nbInputs, const Dims *outputDims, int32_t nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int32_t maxBatchSize)=0
Configure the layer with input and output data types.
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleOnnxMnistCoordConvAC.cpp:204
Definition: parserUtils.h:35
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: cropAndResizePlugin.cpp:160
def preprocess(dynamic_graph)
Definition: preprocess.py:20
virtual void setVerbosityLevel(Verbosity)=0
Set to specific verbosity Level.
virtual const char * getFullTextFileName() const
Get the File Name of the Network Description as a Text File, including the weights.
Definition: parserOnnxConfig.h:120
bool processInput(const samplesCommon::BufferManager &buffers, const std::string &inputTensorName, int inputFileIdx) const
Reads the input and mean data, preprocesses, and stores the result in a managed buffer.
Definition: sampleMNIST.cpp:174
def address(self)
Definition: cuda.py:129
GridAnchorBasePluginCreator()
Definition: gridAnchorPlugin.cpp:351
IPluginV2Ext * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: nvFasterRCNNPlugin.cpp:443
Definition: trt/test_loader.py:245
nvinfer1::Dims loadShape(std::ifstream &input)
Loads shape from weights file.
Definition: sampleMLP.cpp:396
int read(std::vector< Segment_t > &samples, std::shared_ptr< std::istream > input, int samplesToRead=1)
Definition: bleuScoreWriter.cpp:34
void setPluginNamespace(const char *libNamespace) override
Set the namespace that this plugin object belongs to.
Definition: pyramidROIAlignPlugin.cpp:134
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: generateDetectionPlugin.cpp:335
virtual ~IUffParser()
Definition: NvUffParser.h:231
def __init__(self, serialized_engine)
Definition: backend/trt/loader.py:459
String<->Id bijection storage.
Definition: vocabulary.h:34
def test_set_severity(self, sev)
Definition: tf/test_loader.py:30
TypedHostMemory< uint16_t, DataType::kHALF > HalfMemory
Definition: samples/common/common.h:355
static std::ostream & severityOstream(Severity severity)
Definition: pluginLogging.h:162
def test_quant_different_prec(self, verbose)
Definition: quant_rnn_test.py:479
def test_can_build_with_parser_owning(self)
Definition: trt/test_loader.py:246
half operator++(int)
Postfix increment.
Definition: samples/common/half.h:1537
SampleUffSSDParams mParams
The parameters for the sample.
Definition: sampleUffSSD.cpp:91
uint32_t scale_bmm2
Definition: fused_multihead_attention_v2.h:58
std::string magic
Definition: samples/common/common.h:662
float iouThreshold
Definition: NvInferPluginUtils.h:251
handle
Definition: cuda.py:34
int open_ro()
Definition: sampleMovieLensMPS.cpp:131
virtual int32_t getBindingIndex(const char *name) const noexcept=0
Retrieve the binding index for a named tensor.
def reset_parameters(self)
Definition: modules/quant_rnn.py:356
def forward(self, input, hx=None)
Definition: modules/quant_rnn.py:446
Parser modelToNetwork(const ModelOptions &model, nvinfer1::INetworkDefinition &network, std::ostream &err)
Generate a network definition for a given model.
Definition: sampleEngines.cpp:70
def test_simple_default_args(self)
Definition: quant_modules_test.py:34
int mNumLayers
Definition: lstmEncoder.h:54
def test_can_override_shape(self)
Definition: test_data_loader.py:29
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleGoogleNet.cpp:81
virtual Verbosity getVerbosityLevel() const =0
Get the Verbosity Level.
def extra_repr(self)
Definition: modules/quant_rnn.py:230
int mBackgroundLabel
Definition: detectionLayerPlugin.h:93
float outEnd
Definition: sampleReporting.h:99
half nexttoward(half from, long double to)
Next representable value.
Definition: ieee_half.h:2694
int mKeepTopK
Definition: multilevelProposeROIPlugin.h:140
def test_against_unquantized(self)
Definition: quant_conv_transposed_test.py:211
input_node
Definition: test_ir.py:124
def test_with_no_replace_list(self)
Definition: quant_modules_test.py:45
DeviceType
The device that this layer/network will execute on.
Definition: NvInferRuntime.h:706
def find_in_dict(name, mapping, index=None)
Definition: Polygraphy/polygraphy/util/misc.py:36
def __init__(self, seed=None, iterations=None, input_metadata=None, int_range=None, float_range=None)
Definition: comparator/data_loader.py:48
constexpr float defaultPercentile
Definition: sampleOptions.h:53
bias_hh
Definition: modules/quant_rnn.py:350
void writePPMFileWithBBox(const std::string &filename, PPM< C, H, W > &ppm, const BBox &bbox)
Definition: samples/common/common.h:699
reads sequences of data from input stream
Definition: textReader.h:33
ErrorCode
Error codes that can be returned by TensorRT during execution.
Definition: NvInferRuntimeCommon.h:1116
def test_examples(example_dir, artifacts)
Definition: onnx-graphsurgeon/tests/test_examples.py:88
nvinfer1::Dims mOutputDims
The dimensions of the output to the network.
Definition: sampleOnnxMnistCoordConvAC.cpp:78
SampleINT8APIPreprocessing mPreproc
Definition: sampleINT8API.cpp:61
@ kBIDIRECTION
Network iterates from first to last and vice versa and outputs concatenated.
virtual void setPluginFactoryExt(IPluginFactoryExt *factory)=0
Set the IPluginFactoryExt used to create the user defined pluginExts.
static std::vector< PluginField > mPluginAttributes
Definition: generateDetectionPlugin.h:135
def __init__(self, graph)
Definition: graph.py:45
enable< expr, T, U >::type operator+(T x, U y)
Add halfs.
Definition: ieee_half.h:2172
bytes
Definition: Polygraphy/polygraphy/util/misc.py:180
def test_setitem_out_of_bounds(self, run_results)
Definition: comparator/test_struct.py:79
virtual float getDynamicRangeMin() const =0
Get minimum of dynamic range.
const std::string mLayerName
Definition: embLayerNormVarSeqlenPlugin.h:85
virtual void setLocation(TensorLocation location)=0
Set the storage location of a tensor.
const char * getPluginName() const override
Return the plugin name.
Definition: reorgPlugin.cpp:190
def test_range(self)
Definition: calibrator_test.py:298
def __init__(self, tacotron2)
Definition: export_tacotron2_onnx.py:64
std::string mNameSpace
Definition: proposalLayerPlugin.h:113
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: detectionLayerPlugin.cpp:158
int mUnkId
Definition: vocabulary.h:78
def init_decoder_outputs(memory, memory_lengths)
Definition: inference_trt.py:93
def items(self)
Definition: comparator/struct.py:60
def severity(self)
Definition: Polygraphy/polygraphy/logger/logger.py:139
def build_default_profile(builder, network, default_shape_value=None)
Definition: trt/util.py:280
CUresult cuModuleUnload(CUmodule hmod) const
Definition: cudaDriverWrapper.cpp:86
void post()
Definition: sampleMovieLensMPS.cpp:99
T type
Definition: ieee_half.h:382
const char * getPluginVersion() const override
Return the plugin version.
Definition: gridAnchorPlugin.cpp:284
unsigned char fused_multihead_attention_fp16_64_64_kernel_sm80_cu_o[]
Definition: fused_multihead_attention_fp16_64_64_kernel.sm80.cpp:19
Writes the default algorithm choices made by TensorRT into a file.
Definition: sampleAlgorithmSelector.cpp:51
def convert_to_unicode(text)
Definition: tokenization.py:77
float visualThreshold
The minimum score threshold to consider a detection.
Definition: sampleSSD.cpp:52
const char * getPluginName() const override
Return the plugin name.
Definition: embLayerNormVarSeqlenPlugin.cpp:425
def get_input_metadata(self)
Definition: backend/trt/runner.py:53
void parseMovieLensData()
Parses the MovieLens dataset and populates the SampleMovieLensParams data structure.
Definition: sampleMovieLens.cpp:423
const char * getPluginVersion() const override
Return the plugin version.
Definition: nvFasterRCNNPlugin.cpp:344
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: reorgPlugin.cpp:154
const char * getPluginType() const override
Return the plugin type.
Definition: nmsPlugin.cpp:148
Definition: quant_pooling.py:49
@ kDIMS
nvinfer1::Dims field type.
def __init__(self, name=None, prefix=None)
Definition: backend/base/runner.py:29
const char * getPluginVersion() const override
Return the plugin version.
Definition: batchedNMSPlugin.cpp:365
TrtHostBuffer mHostBuffer
Definition: sampleDevice.h:350
virtual IShuffleLayer * addShuffle(ITensor &input)=0
Add a shuffle layer to the network.
ProfilingVerbosity
List of verbosity levels of layer information exposed in NVTX annotations.
Definition: NvInfer.h:6415
Definition: samples/common/common.h:807
int h
Definition: samples/common/common.h:663
def test_no_quant(self)
Definition: quant_conv_transposed_test.py:240
std::array< std::unique_ptr< TrtCudaEvent >, static_cast< int >(EventType::kNUM)> MultiEvent
Definition: sampleInference.cpp:273
The SampleGoogleNet class implements the GoogleNet sample.
Definition: sampleGoogleNet.cpp:46
float mScale
Definition: resizeNearestPlugin.h:90
def __init__(self, model, do_shape_inference=None, outputs=None, exclude_outputs=None)
Definition: backend/onnx/loader.py:108
void serializeFromDevice(char *&hostBuffer, Weights deviceWeights) const
Definition: gridAnchorPlugin.cpp:262
std::mutex mutex
Definition: sampleInference.cpp:168
def test_inference(encoder, decoder_iter, postnet)
Definition: export_tacotron2_onnx.py:208
n_mel_channels
Definition: demo/Tacotron2/tacotron2/model.py:242
Definition: samples/common/logging.h:34
max_batch_size
Definition: backend/trt_legacy.py:224
wordpiece_tokenizer
Definition: tokenization.py:192
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: flattenConcat.cpp:215
def test_properties(self)
Definition: tensor_quantizer_test.py:149
ProposalLayer(int prenms_topk, int keep_topk, float iou_threshold, const nvinfer1::Dims &image_size)
Definition: proposalLayerPlugin.cpp:103
@ kSELU
Selu activation: x>0 ? beta * x : beta * (alpha*exp(x) - alpha)
nvinfer1::IPluginV2DynamicExt * clone() const override
Clone the plugin object.
Definition: instanceNormalizationPlugin.cpp:254
static PluginFieldCollection mFC
Definition: flattenConcat.h:126
std::atomic< int32_t > failCount
Definition: sampleMovieLensMPS.cpp:186
input_size
Definition: modules/quant_rnn.py:387
@ kVERBOSE
Register layer names in NVTX message field and register layer detail in NVTX JSON payload field.
def test_import_variable_tensor(self)
Definition: test_importers.py:33
def test_quant_input_hidden(self, verbose)
Definition: quant_rnn_test.py:417
const char * getPluginVersion() const override
Return the plugin version.
Definition: detectionLayerPlugin.cpp:52
std::vector< nvinfer1::Weights > mGateKernelWeights
Definition: lstmDecoder.h:47
std::string mNameSpace
Definition: detectionLayerPlugin.h:105
int postNmsTopN
Definition: sampleUffFasterRCNN.cpp:78
Logger & report(Severity severity, const char *msg)
Definition: demo/BERT/infer_c/logging.h:47
void setPluginNamespace(const char *libNamespace) override
Set the namespace that this plugin object belongs to.
Definition: generateDetectionPlugin.cpp:180
query_layer
Definition: demo/Tacotron2/tacotron2/model.py:51
constexpr int defaultSleep
Definition: sampleOptions.h:49
~DetectionLayer() override=default
def test_loader_explicit_precision(self)
Definition: trt/test_loader.py:97
expr(float f)
Conversion constructor.
Definition: ieee_half.h:357
def _clean_text(text, cleaner_names)
Definition: demo/Tacotron2/tacotron2/text/__init__.py:56
def test_pop(self, field_names)
Definition: test_ir.py:261
unsigned char fused_multihead_attention_v2_fp16_96_64_kernel_sm75_cubin[]
Definition: fused_multihead_attention_v2_fp16_96_64_kernel.sm75.cpp:19
int mMaxBatchSize
Definition: multilevelProposeROIPlugin.h:101
def override_dynamic_shape(shape)
Definition: Polygraphy/polygraphy/util/misc.py:153
static size_t sizeOfCaffeType(trtcaffe::Type type)
Definition: caffeWeightFactory.cpp:265
def collect(self, x)
Definition: tools/pytorch-quantization/pytorch_quantization/calib/calibrator.py:36
def __getitem__(self, key)
Definition: comparator/struct.py:98
FreeFunc freeFn
Definition: buffers.h:176
const Dims PluginFormat
Definition: NvInferRuntimeCommon.h:792
bool mInitialized
Definition: caffeWeightFactory.h:63
std::map< std::string, void * > mInputDeviceBuffers
Definition: sampleEngines.cpp:186
def test_fake_quant_per_channel(self)
Definition: quant_instancenorm_test.py:182
def test_skip_zeros(self, verbose)
Definition: calibrator_test.py:124
def test_set_inputs_updates_old_inputs(self)
Definition: test_ir.py:55
Weights copyToDevice(const void *hostData, size_t count)
Definition: normalizePlugin.cpp:144
void setShouldLog(bool shouldLog)
Definition: samples/common/logging.h:96
BinaryProtoBlob(void *memory, nvinfer1::DataType type, nvinfer1::DimsNCHW dimensions)
Definition: binaryProtoBlob.h:29
node
Definition: test_ir.py:224
virtual ~IPluginRegistry() noexcept
Definition: NvInferRuntimeCommon.h:980
def test_multiple_runners(self)
Definition: test_comparator.py:67
Definition: backend/tf/loader.py:270
static std::basic_ostream< charT, traits > & write(std::basic_ostream< charT, traits > &out, float arg)
Output implementation.
Definition: samples/common/half.h:1626
def apply_quant_modules(self)
Definition: quant_modules.py:94
def main()
Definition: export_tacotron2_onnx.py:280
cudnnTensorDescriptor_t _x_desc
Definition: instanceNormalizationPlugin.h:98
int _nx
Definition: split.h:42
float mIOUThreshold
Definition: detectionLayerPlugin.h:130
int mW
Definition: priorBoxPlugin.h:93
matmul_q_input_quantizer
Definition: quant_bert.py:73
virtual ~SampleErrorRecorder() noexcept
Definition: ErrorRecorder.h:45
virtual void serialize(void *buffer) override
Serialize the layer.
Definition: samples/opensource/samplePlugin/fcPlugin.h:210
Definition: Tacotron2/inference.py:164
nvinfer1::Dims dims
Definition: sampleReformatFreeIO.cpp:183
int stride
Definition: reorgPlugin.h:111
int mNumUnits
Definition: lstmEncoder.h:55
const char * PRIOR_BOX_PLUGIN_NAME
Definition: priorBoxPlugin.cpp:33
Definition: nmsPlugin.h:94
Definition: Polygraphy/polygraphy/common/exception.py:16
static PluginFieldCollection mFC
Definition: reorgPlugin.h:110
nvinfer1::Weights convertRNNWeights(nvinfer1::Weights input, int dataSize)
Converts RNN weights from TensorFlow's format to TensorRT's format.
Definition: sampleCharRNN.cpp:441
@ DATA_TYPE_E8M10
Definition: fused_multihead_attention_common.h:25
bool teardown()
Cleans up any state created in the sample class.
Definition: sampleUffFasterRCNN.cpp:369
virtual void addToModel(nvinfer1::INetworkDefinition *network, nvinfer1::ITensor *input, nvinfer1::ITensor **output)=0
add the embedding vector calculation to the network
Definition: optim_helper_test.py:51
def export_onnx(parser, args)
Definition: export_waveglow_onnx.py:156
bert::cuda_unique_ptr< void > mGammaDev
Definition: skipLayerNormInt8InterleavedPlugin.h:74
def test_input_variable_bits(self)
Definition: quant_pooling_test.py:260
RNNInputMode
Enumerates the RNN input modes that may occur with an RNN layer.
Definition: NvInfer.h:2841
void run(const void *const *inputBuffers, int warmUps, int iterations)
Definition: bert_infer.h:219
def test_can_compare_bool(self)
Definition: test_compare.py:23
float timeMs
Definition: sampleReporting.h:161
def test_per_tensor_scale(self)
Definition: tensor_quant_test.py:46
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleGoogleNet.cpp:49
int getTRTOptionIndex(const std::vector< TRTOption > &options, const TRTOption &opt)
getTRTOptionIndex returns the index of a TRTOption in a vector of TRTOptions, -1 if not found.
Definition: getOptions.cpp:64
def test_raise(self)
Definition: quant_pooling_test.py:289
static PluginFieldCollection mFC
Definition: gridAnchorPlugin.h:123
~GenerateDetection() override=default
SampleUniquePtr< nvcaffeparser1::IBinaryProtoBlob > mMeanBlob
Definition: sampleAlgorithmSelector.cpp:364
int mCurrentBatch
Definition: sampleEngines.cpp:184
use_fc2_gemm
Definition: builder.py:99
int coords
Definition: regionPlugin.h:94
@ kBOOL
8-bit boolean. 0 = false, 1 = true, other values undefined.
SampleUniquePtr< T > makeUnique(T *t)
Definition: sampleDynamicReshape.cpp:94
TestResult
Represents the state of a given test.
Definition: pluginLogging.h:239
unsigned int fused_multihead_attention_v2_fp16_96_64_kernel_sm75_cubin_len
Definition: fused_multihead_attention_v2_fp16_96_64_kernel.sm75.cpp:2852
def __contains__(self, val)
Definition: comparator/struct.py:122
static expr asinh(float arg)
Hyperbolic area sine implementation.
Definition: samples/common/half.h:1967
int mNbClasses
Definition: generateDetectionPlugin.h:131
TrtUniquePtr< nvonnxparser::IParser > onnxParser
Definition: sampleEngines.h:37
int32_t featureStride
Definition: NvInferPluginUtils.h:120
__host__ __device__ __forceinline__ int get() const
Definition: reducedMathPlugin.h:85
def __init__(self, output_size, **kwargs)
Definition: quant_pooling.py:119
Definition: backend/trt/runner.py:1
float ** mDeviceBboxes
Definition: multilevelProposeROIPlugin.h:110
void addToModel(nvinfer1::INetworkDefinition *network, nvinfer1::ITensor *inputEmbeddedData, nvinfer1::ITensor **inputStates, nvinfer1::ITensor **outputData, nvinfer1::ITensor **outputStates) override
add the memory, cell, and hidden states to the network
Definition: lstmDecoder.cpp:64
Definition: onnx_exporter.py:1
n_convs
Definition: demo/Tacotron2/tacotron2/model.py:159
def parse_decoder_outputs(self, mel_outputs, gate_outputs, alignments)
Definition: demo/Tacotron2/tacotron2/model.py:361
std::vector< std::vector< float > > mAnchorBoxesHost
Definition: multilevelProposeROIPlugin.h:106
std::vector< size_t > argMagnitudeSort(Iter begin, Iter end)
Return vector of indices that puts magnitudes of sequence in descending order.
Definition: samples/common/common.h:399
def binding(self)
Definition: infer.py:36
static const int MAX_PRE_NMS_RESULTS
Definition: tlt_mrcnn_config.h:76
Definition: sampleOptions.h:69
def test_items(self, run_results)
Definition: comparator/test_struct.py:28
Definition: trt/test_loader.py:120
tf_rep
Definition: backend/onnx/runner.py:43
static const int NUM_CLASSES
Definition: tlt_mrcnn_config.h:56
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: nvFasterRCNNPlugin.cpp:214
def invoke(func, *args, **kwargs)
Definition: func.py:101
Definition: test_exporters.py:33
bool update()
Definition: BatchStream.h:295
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: gridAnchorPlugin.cpp:374
def __del__(self)
Definition: backend/base/runner.py:156
SampleAlgorithmSelector(const samplesCommon::CaffeSampleParams ¶ms)
Definition: sampleAlgorithmSelector.cpp:318
Definition: backend/trt_legacy.py:179
SLPEmbedder(ComponentWeights::ptr weights)
Definition: slpEmbedder.cpp:25
def _remove_commas(m)
Definition: numbers.py:31
def test_mark_shape_outputs(self, reshape_network)
Definition: trt/test_loader.py:145
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: regionPlugin.cpp:419
constexpr int32_t EnumMax< PaddingMode >()
Maximum number of elements in PaddingMode enum.
Definition: NvInfer.h:1182
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: lReluPlugin.cpp:58
def dict(self)
Definition: tensor_quant.py:174
Definition: quant_instancenorm.py:65
Definition: comparator/comparator.py:1
int getElememtSize()
Definition: sampleReformatFreeIO.cpp:123
float mScoreThreshold
Definition: detectionLayerPlugin.h:96
owns_engine
Definition: backend/trt/runner.py:80
virtual ~IExprBuilder()
Definition: NvInferRuntime.h:367
size_t _d_bytes
Definition: instanceNormalizationPlugin.h:96
def test_default_tol_in_map(self, mode)
Definition: test_compare.py:65
nvinfer1::Dims mPredictionOutputDims
The dimensions of the output of the MNIST model.
Definition: sampleDynamicReshape.cpp:81
uint32_t NetworkDefinitionCreationFlags
This bitset is capable of representing one or more NetworkDefinitionCreationFlag flags constructed wi...
Definition: NvInfer.h:6880
def values(self)
Definition: comparator/struct.py:77
int volume(const nvinfer1::Dims &d)
Definition: sampleUtils.h:64
virtual ~IBlobNameToTensor()
Definition: NvCaffeParser.h:59
static const float MASK_THRESHOLD
Definition: tlt_mrcnn_config.h:34
def test_basic(self)
Definition: onnxrt/test_runner.py:37
virtual void destroy()=0
De-allocates any internally allocated memory.
long lround(half arg)
Nearest integer.
Definition: ieee_half.h:2579
def has_colors()
Definition: Polygraphy/polygraphy/logger/logger.py:29
def test_raise(self)
Definition: quant_pooling_test.py:40
timeline_dir
Definition: backend/tf/runner.py:53
float getScore() const
Definition: bleuScoreWriter.cpp:190
ComponentWeights::ptr mWeights
Definition: lstmDecoder.h:46
nvinfer1::DimsHW sortedBBoxDims
Definition: maskRCNNKernels.h:198
def test_can_build_without_parser_non_owning(self, identity_builder_network)
Definition: trt/test_loader.py:252
std::vector< std::string > outputs
Definition: sampleOptions.h:98
Temporary half-precision expression.
Definition: ieee_half.h:353
def test_iadd(self, field_names)
Definition: test_ir.py:301
bool isfinite(half arg)
Check if finite number.
Definition: ieee_half.h:2728
def setup_method(self)
Definition: test_ir.py:122
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: nvFasterRCNNPlugin.cpp:309
int getH(const Dims &d)
Definition: samples/common/common.h:886
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: detectionLayerPlugin.cpp:294
nvinfer1::ProfilingVerbosity nvtxMode
Definition: sampleOptions.h:120
unsigned char fused_multihead_attention_v2_int8_192_64_kernel_cubin[]
Definition: fused_multihead_attention_v2_int8_192_64_kernel.sm72.cpp:19
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
Definition: sampleUffMaskRCNN.cpp:340
bias_hh
Definition: modules/quant_rnn.py:435
An array of field params used as a layer parameter for plugin layers.
Definition: NvUffParser.h:76
Definition: demo/Tacotron2/tacotron2/model.py:122
calculates likelihood and TopK indices for the raw input logits
Definition: likelihood.h:33
virtual bool setWeights(const char *layerName, WeightsRole role, Weights weights)=0
Specify new weights for a layer of given name.
def initializer_is_output_model()
Definition: onnx_models.py:175
def test_o_multiple_outputs(self)
Definition: test_ir.py:211
void copyInputToDeviceAsync(const cudaStream_t &stream=0)
Copy the contents of input host buffers to input device buffers asynchronously.
Definition: buffers.h:411
Dims mOutputDims
Definition: sampleUffPluginV2Ext.cpp:639
std::chrono::time_point< std::chrono::high_resolution_clock > TimePoint
Definition: sampleInference.cpp:160
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleAlgorithmSelector.cpp:561
Half-precision floating point type.
Definition: ieee_half.h:1084
ReportingOptions reporting
Definition: sampleOptions.h:192
_num_bits
Definition: tensor_quantizer.py:74
encoder_embedding_dim
Definition: demo/Tacotron2/tacotron2/model.py:244
linear_layer
Definition: layers.py:26
bool isgreaterequal(half x, half y)
Comparison for greater equal.
Definition: ieee_half.h:2784
virtual void addToModel(nvinfer1::INetworkDefinition *network, nvinfer1::ITensor *inputFromDecoder, nvinfer1::ITensor *context, nvinfer1::ITensor **attentionOutput)=0
add the attention vector calculation to the network
Definition: maskRCNNKernels.h:63
Iteration(int id, const InferenceOptions &inference, nvinfer1::IExecutionContext &context, Bindings &bindings)
Definition: sampleInference.cpp:285
float * maxSize
Definition: NvInferPluginUtils.h:93
bool mOK
Definition: caffeWeightFactory.h:65
void setPluginNamespace(const char *libNamespace) override
Set the namespace that this plugin object belongs to.
Definition: detectionLayerPlugin.cpp:165
Definition: quant_modules.py:1
ICudaEngine * modelToEngine(const ModelOptions &model, const BuildOptions &build, const SystemOptions &sys, std::ostream &err)
Create an engine for a given model.
Definition: sampleEngines.cpp:590
Layer type for plugins.
Definition: NvInfer.h:3421
const char * getPluginVersion() const
Return the plugin version.
Definition: split.h:147
int mTimestepId
Definition: beamSearchPolicy.h:74
const char * getPluginVersion() const override
Return the plugin version.
Definition: flattenConcat.cpp:274
def is_dimension_dynamic(dim)
Shapes.
Definition: Polygraphy/polygraphy/util/misc.py:134
Definition: backend/onnx/runner.py:1
static half fmax(half x, half y)
Definition: samples/common/half.h:2594
encodes single input into output states with LSTM
Definition: lstmDecoder.h:31
int mAnchorsCnt
Definition: proposalLayerPlugin.h:103
Type traits for floating point types.
Definition: ieee_half.h:305
uint32_t scale_bmm1
Definition: fused_multihead_attention.h:81
CUresult(* _cuLinkCreate)(unsigned int, CUjit_option *, void **, CUlinkState *)
Definition: cudaDriverWrapper.h:75
void addToModel(nvinfer1::INetworkDefinition *network, int beamWidth, nvinfer1::ITensor *inputLogits, nvinfer1::ITensor *inputLikelihoods, nvinfer1::ITensor **newCombinedLikelihoods, nvinfer1::ITensor **newRayOptionIndices, nvinfer1::ITensor **newVocabularyIndices) override
add calculation of likelihood and TopK indices to the network
Definition: softmaxLikelihood.cpp:25
Definition: NvInfer.h:4523
virtual int32_t enqueue(int32_t batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream)=0
Execute the layer.
def spectral_de_normalize(self, magnitudes)
Definition: layers.py:74
SplitPluginCreator()
Definition: split.h:138
def test_can_build_with_calibrator(self, identity_builder_network)
Definition: trt/test_loader.py:259
sess
Definition: backend/tf/runner.py:75
def test_calibrator_rechecks_cache_on_reset(self, identity_builder_network)
Definition: test_calibrator.py:128
def test_simple_run(self)
Definition: functional_test.py:37
def convert_1d_to_2d_(glow)
Definition: export_waveglow_onnx.py:83
int s
Definition: fused_multihead_attention.h:79
std::mutex & getMutex()
Definition: pluginLogging.h:156
unsigned int fused_multihead_attention_fp16_128_64_kernel_sm80_cu_o_len
Definition: fused_multihead_attention_fp16_128_64_kernel.sm80.cpp:2283
def volume
Definition: 09_shape_operations_with_the_layer_api/generate.py:59
@ kWAIVED
The test was waived.
Definition: common/struct.py:1
void deserialize(void const *serialData, size_t serialLength)
Definition: split.h:53
LogStream< ILogger::Severity::kERROR > gLogError
Definition: checkMacrosPlugin.cpp:48
Definition: sampleUffPluginV2Ext.cpp:172
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: batchedNMSPlugin.cpp:284
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: batchedNMSPlugin.cpp:511
def test_basic(self)
Definition: trt/test_runner.py:42
def __exit__(self, exc_type, exc_value, traceback)
Definition: Polygraphy/tests/test_examples.py:77
def scale_amax(self)
Definition: tensor_quant.py:132
const char * getPluginVersion() const override
Return the plugin version.
Definition: coordConvACPlugin.cpp:134
graph
Definition: backend/tf/loader.py:405
InstanceNormalizationPlugin()=delete
Definition: backend/onnxrt/runner.py:25
def batch_to_gpu(batch)
Definition: tacotron2/data_function.py:132
nvinfer1::DimsHW argMaxBboxDims
Definition: maskRCNNKernels.h:165
void stop()
Definition: samples/common/common.h:849
static const Dims4 MODEL_MASK_SHAPE
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:175
def test_from_to_dict(self)
Definition: tensor_quant_test.py:325
def test_extend_can_modify_rv_objects(self)
Definition: test_func.py:118
nvinfer1::Dims mInputDims
The dimensions of the input to the network.
Definition: sampleOnnxMnistCoordConvAC.cpp:77
@ kRAGGED_SOFTMAX
Ragged softmax layer.
use_xla
Definition: backend/tf/loader.py:371
@ kCLIP
Clip activation: max(alpha, min(beta, x))
CUfunction mDeviceFunction
Definition: fused_multihead_attention.h:285
size_t mWordVocabSize
Definition: embLayerNormVarSeqlenPlugin.h:94
InferenceTime operator+(const InferenceTime &a, const InferenceTime &b)
Definition: sampleReporting.h:102
nvinfer1::DimsHW argMaxScoreDims
Definition: maskRCNNKernels.h:97
def test_int8(self, identity_builder_network, flag)
Definition: trt/test_loader.py:218
const void * data
Definition: NvUffParser.h:80
std::vector< std::string > classNames
Definition: sampleUffFasterRCNN.cpp:76
def test_i(self)
Definition: test_ir.py:190
std::string longName
Option name in long (double hyphen) form (i.e. –foo, –bar)
Definition: getOptions.h:37
_fake_quant
Definition: tensor_quantizer.py:75
def debug(self, message, mode=LogMode.EACH)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:227
virtual DataType getDataType() const =0
Return DataType of the input/output of algorithm.
static PluginFieldCollection mFC
Definition: instanceNormalizationPlugin.h:120
def test_no_quant(self)
Definition: quant_conv_test.py:51
std::string mSuffix
Batch file name suffix.
Definition: BatchStream.h:384
logger
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:43
Definition: cropAndResizePlugin.h:36
int32_t outputClsSize
The number of output classes.
Definition: sampleUffSSD.cpp:52
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: generateDetectionPlugin.cpp:329
Definition: bboxUtils.h:40
def check_profile(profile)
Definition: trt/util.py:274
output_tensor
Definition: test_ir.py:137
virtual const char * getModelFileName() const =0
Get the Model FileName.
static constexpr int32_t kSM_XAVIER
Definition: qkvToContextInt8InterleavedPlugin.h:29
std::vector< void * > & getTmpAllocs()
Definition: caffeWeightFactory.cpp:88
def get_opset(onnx.ModelProto model)
Definition: onnx_importer.py:71
bool getShapesInference(Arguments &arguments, std::unordered_map< std::string, std::vector< int >> &shapes, const char *argument)
Definition: sampleOptions.cpp:258
Plugin factory for deserialization.
Definition: NvInferRuntime.h:966
const char * getPluginType() const override
Return the plugin type.
Definition: priorBoxPlugin.cpp:234
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: debugUtil.cpp:75
static bool isnormal(half arg)
Classification implementation.
Definition: samples/common/half.h:2369
def reset_parameters(self)
Definition: modules/quant_rnn.py:402
virtual void setAllowedFormats(TensorFormats formats)=0
Set allowed formats for this tensor.
void serialize(void *buffer) const override
Serialize the layer.
Definition: sampleUffPluginV2Ext.cpp:475
def test_no_quant_input_hidden(self, verbose)
Definition: quant_rnn_test.py:310
LReLU(float negSlope)
Definition: lReluPlugin.cpp:32
def main()
Definition: train.py:326
bool skipTransfers
Definition: sampleOptions.h:155
def test_simple_run(self)
Definition: calibrator_test.py:36
int inferTypeToBytes(nvinfer1::DataType t)
Definition: trtUtil.cpp:25
nvinfer1::Dims mImageSize
Definition: proposalLayerPlugin.h:109
nvinfer1::ILayer * parseELU(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &, BlobNameToTensor &tensors)
Definition: parseELU.cpp:23
def english_cleaners(text)
Definition: cleaners.py:98
Managed CUDA event.
Definition: sampleDevice.h:107
std::vector< LayerProfile > mLayers
Definition: sampleReporting.h:188
static void help(std::ostream &out)
Definition: sampleOptions.cpp:1155
def test_basic(self)
Definition: onnx/test_loader.py:36
PluginType
The type values for the various plugins.
Definition: NvInferPluginUtils.h:37
static half trunc(half arg)
Truncation implementation.
Definition: samples/common/half.h:2094
std::shared_ptr< ComponentWeights > ptr
Definition: componentWeights.h:34
int s
Definition: fused_multihead_attention_v2.h:56
Definition: tools/Polygraphy/polygraphy/common/__init__.py:1
const char * getPluginType() const override
Return the plugin type.
Definition: qkvToContextInt8InterleavedPlugin.cpp:167
Definition: fused_multihead_attention.h:48
Definition: infer_c.cpp:23
unsigned char type
Definition: samples/common/half.h:375
const char * getPluginVersion() const override
Return the plugin version.
Definition: gridAnchorPlugin.cpp:369
size_t mCopySize
Definition: batchTilePlugin.h:80
NMSParameters param
Definition: batchedNMSPlugin.h:112
def get_input_metadata(self)
Definition: backend/tf/runner.py:69
float eps
Definition: normalizePlugin.h:100
virtual int32_t getNbInputs() const =0
Get the number of inputs in the network.
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: cropAndResizePlugin.cpp:268
prenet_dim
Definition: demo/Tacotron2/tacotron2/model.py:247
float mDqProbs
Definition: qkvToContextInt8InterleavedPlugin.h:90
virtual bool isConstant() const =0
Return true if expression is a build-time constant.
def test_license(self)
Definition: license_test.py:28
Definition: sampleOptions.h:146
def test_weight_fake_quant_per_channel(self)
Definition: quant_conv_transposed_test.py:88
const char * RPROI_PLUGIN_NAME
Definition: nvFasterRCNNPlugin.cpp:31
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleMNIST.cpp:94
TacticSource
List of tactic sources for TensorRT.
Definition: NvInfer.h:6437
unsigned char fused_multihead_attention_v2_fp16_96_64_kernel_sm80_cubin[]
Definition: fused_multihead_attention_v2_fp16_96_64_kernel.sm80.cpp:19
IPluginV2Ext * deserializePlugin(const char *name, const void *data, size_t length) override
Called during deserialization of plugin layer.
Definition: multilevelCropAndResizePlugin.cpp:84
int64_t qkv_stride_in_bytes
Definition: fused_multihead_attention.h:58
def test_per_channel_scale(self)
Definition: tensor_quant_test.py:57
def test_fake_quant_input(self)
Definition: quant_conv_test.py:327
static const int POST_NMS_ROIS_INFERENCE
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:83
def test_layerwise(self)
Definition: onnx/test_loader.py:54
A divUp(A m, B n)
Definition: parserUtils.h:135
int32_t nbCalBatches
The number of batches for calibration.
Definition: sampleUffSSD.cpp:54
void constructNetwork(SampleUniquePtr< nvuffparser::IUffParser > &parser, SampleUniquePtr< nvinfer1::INetworkDefinition > &network)
Parses a Uff model for MNIST and creates a TensorRT network.
Definition: sampleUffMNIST.cpp:173
static PluginFieldCollection mFC
Definition: proposalLayerPlugin.h:134
virtual int getAttentionSize()=0
get the size of the attention vector
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: proposalPlugin.cpp:283
bert::cuda_unique_ptr< float > mBetaDev
Definition: embLayerNormVarSeqlenPlugin.h:89
InferenceTrace & operator=(const InferenceTrace &)=default
int mInputChannelCount
Definition: multiplicativeAlignment.h:53
Definition: tacotron2/data_function.py:26
void dumpBindingValues(int binding, std::ostream &os, const std::string &separator=" ") const
Definition: sampleUtils.h:378
Definition: NvInfer.h:4622
def test_list_as_data_loader(self)
Definition: test_comparator.py:40
ComponentWeights::ptr mWeights
Definition: slpAttention.h:47
@ kMAX
This is used to set or get the maximum permitted value for dynamic dimensions etc.
std::vector< samplesCommon::PPM< kIMG_CHANNELS, kIMG_H, kIMG_W > > mPPMs
PPMs of test images.
Definition: sampleFasterRCNN.cpp:90
void reset(int firstBatch)
Definition: frcnnUtils.h:244
const char * HIDDEN_OUT_BLOB_NAME
Definition: sampleCharRNN.cpp:90
int64_t volume(const nvinfer1::Dims &d)
Definition: samples/common/common.h:627
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: resizeNearestPlugin.cpp:228
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: instanceNormalizationPlugin.cpp:267
static double erf(double arg)
Definition: samples/common/half.h:2476
@ kSCALED_TANH
Scaled tanh activation: alpha*tanh(beta*x)
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: generateDetectionPlugin.cpp:249
float x1
Definition: samples/common/common.h:669
Linear attention calculation.
Definition: slpAttention.h:34
static void help(std::ostream &out)
Definition: sampleOptions.cpp:1229
static expr remquo(float x, float y, int *quo)
Remainder implementation.
Definition: samples/common/half.h:1688
unsigned char fused_multihead_attention_v2_int8_384_64_kernel_sm80_cubin[]
Definition: fused_multihead_attention_v2_int8_384_64_kernel.sm80.cpp:19
A resize layer in a network definition.
Definition: NvInfer.h:4364
def deactivate_impl(self)
Definition: backend/trt_legacy.py:325
def __exit__(self, exc_type, exc_value, traceback)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:36
std::vector< float > anchorsScales
Definition: nvFasterRCNNPlugin.h:124
Tag type for binary construction.
Definition: ieee_half.h:346
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:257
DeviceBuffer(size_t elementCount)
Definition: deviceBuffer.h:32
void setAllTensorScales(INetworkDefinition *network, float inScales=2.0f, float outScales=4.0f)
Definition: samples/common/common.h:520
def resnet18()
Definition: tools/pytorch-quantization/tests/fixtures/models.py:62
virtual void serialize(void *buffer) const =0
Serialize the layer.
static expr log1p(float arg)
Logarithm implementation.
Definition: samples/common/half.h:1815
unsigned int fused_multihead_attention_v2_fp16_384_64_kernel_sm80_cubin_len
Definition: fused_multihead_attention_v2_fp16_384_64_kernel.sm80.cpp:6003
max_workspace_size
Definition: backend/trt_legacy.py:216
vocab
Definition: tokenization.py:188
def __init__(self, model, name=None)
Definition: backend/cntk/runner.py:26
def str_from_layer(prefix, index, name, op, input_info, output_info)
Definition: Polygraphy/polygraphy/util/misc.py:405
def test_simple_run(self)
Definition: clip_test.py:38
void serialize(void *buffer) const override
Serialize the layer.
Definition: detectionLayerPlugin.cpp:180
void setClipParam(bool clip)
Definition: batchedNMSPlugin.cpp:449
void check_valid_inputs(const nvinfer1::Dims *inputs, int nbInputDims)
Definition: multilevelProposeROIPlugin.cpp:292
static half round(half arg)
Nearest integer implementation.
Definition: samples/common/half.h:2102
clip_value_min
Definition: clip.py:45
Definition: samples/opensource/samplePlugin/fcPlugin.h:36
def __call__(self)
Definition: backend/tf/loader.py:122
def initialize_decoder_states(self, memory)
Definition: demo/Tacotron2/tacotron2/model.py:304
nvinfer1::DataType mType
Definition: detectionLayerPlugin.h:102
void configurePlugin(const DynamicPluginTensorDesc *in, int nbInputs, const DynamicPluginTensorDesc *out, int nbOutputs) override
Definition: batchedNMSPlugin.cpp:305
virtual ~IAlgorithm()
Definition: NvInfer.h:6294
def __enter__(self)
Definition: Polygraphy/polygraphy/util/misc.py:445
Definition: trt/test_loader.py:105
def test_fake_quant_per_channel_other_prec(self)
Definition: quant_conv_test.py:164
uint32_t TensorFormats
It is capable of representing one or more TensorFormat by binary OR operations, e....
Definition: NvInferRuntimeCommon.h:210
size_t mLd
Definition: skipLayerNormInt8InterleavedPlugin.h:76
def expand_numbers(text)
Definition: cleaners.py:67
void scaleShiftChannelsInplace(T *inOut, const int B, const int C, const int channelVolume, const float *beta, const float *gamma, cudaStream_t stream)
std::shared_ptr< BeamSearchPolicy > ptr
Definition: beamSearchPolicy.h:36
old_indent
Definition: Polygraphy/polygraphy/logger/logger.py:46
static std::vector< PluginField > mPluginAttributes
Definition: nvFasterRCNNPlugin.h:125
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleFasterRCNN.cpp:205
def test_test_input_fake_per_tensor(self)
Definition: quant_linear_test.py:89
std::string uffFileName
Definition: sampleUffMaskRCNN.cpp:306
def test_io_cannot_be_sync_list_on_assign(self)
Definition: test_ir.py:789
def assert_equal(self, Graph graph)
Definition: onnx_models.py:45
def get_loss_function(loss_function, sigma=1.0)
Definition: loss_functions.py:23
CUresult cuLaunchCooperativeKernel(CUfunction f, unsigned int gridDimX, unsigned int gridDimY, unsigned int gridDimZ, unsigned int blockDimX, unsigned int blockDimY, unsigned int blockDimZ, unsigned int sharedMemBytes, CUstream hStream, void **kernelParams) const
Definition: cudaDriverWrapper.cpp:124
Definition: fused_multihead_attention.h:189
sample::Logger::TestResult teardown()
Used to clean up any state created in the sample class.
Definition: sampleINT8API.cpp:638
/brief Struct to maintain command-line arguments.
Definition: argsParser.h:79
def load_code_blocks_from_readme(readme)
Definition: Polygraphy/tests/test_examples.py:33
Definition: test_cuda.py:101
Definition: demo/Tacotron2/waveglow/model.py:157
bool isinf(half arg)
Check for infinity.
Definition: ieee_half.h:2736
line_info
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:107
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: nmsPlugin.cpp:78
float mBboxMinSize
Definition: proposalPlugin.h:115
half & operator++()
Prefix increment.
Definition: samples/common/half.h:1523
static DataType getParamWordType(DataType cfgType)
Definition: skipLayerNormInt8InterleavedPlugin.cpp:53
@ DATA_TYPE_BOOL
Definition: fused_multihead_attention_common.h:24
static PluginFieldCollection mFC
Definition: regionPlugin.h:120
std::shared_ptr< Vocabulary > ptr
Definition: vocabulary.h:37
def test_tensors_includes_non_node_tensors(self)
Definition: test_ir.py:579
bool supportsFormatCombination(int pos, const PluginTensorDesc *inOut, int nbInputs, int nbOutputs) override
Definition: batchedNMSPlugin.cpp:336
expected
Definition: test_misc.py:39
std::vector< IOFormat > inputFormats
Definition: sampleOptions.h:125
const char * getPluginType() const override
Return the plugin type.
Definition: batchedNMSPlugin.cpp:355
static const struct bert::FusedMultiHeadAttentionKernelMetaInfoV1 sMhaKernelMetaInfos[]
Definition: sampleMovieLensMPS.cpp:80
outputs
Definition: tensor.py:166
def mark_layerwise(network)
Definition: trt/util.py:225
level
Definition: graph.py:265
unsigned char fused_multihead_attention_v2_fp16_384_64_kernel_sm80_cubin[]
Definition: fused_multihead_attention_v2_fp16_384_64_kernel.sm80.cpp:19
def forward(self, input)
Definition: quant_conv.py:117
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: proposalPlugin.cpp:188
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: generateDetectionPlugin.cpp:190
def _compute_amax_percentile(self, percentile)
Definition: histogram.py:196
void destroy() override
Destroy the plugin object.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:248
Definition: NvInfer.h:2674
std::string outputClsName
Definition: sampleUffFasterRCNN.cpp:63
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: pyramidROIAlignPlugin.cpp:254
def error(self, message, mode=LogMode.EACH)
Definition: Polygraphy/polygraphy/logger/logger.py:282
ICudaEngine * networkToEngine(const BuildOptions &build, const SystemOptions &sys, IBuilder &builder, INetworkDefinition &network, std::ostream &err)
Create an engine for a network defintion.
Definition: sampleEngines.cpp:301
int32_t numClasses
Definition: NvInferPluginUtils.h:184
load_mel_from_disk
Definition: tacotron2/data_function.py:37
void destroy() override
Destroy the plugin object.
Definition: qkvToContextInt8InterleavedPlugin.cpp:206
bool verifyOutput(const samplesCommon::BufferManager &buffers)
Classifies digits and verify result.
Definition: sampleMNISTAPI.cpp:355
def free(self)
Definition: cuda.py:139
bool next()
Definition: frcnnUtils.h:253
def __init__(self, bert_config_path, use_fp16, use_int8, use_strict, use_fc2_gemm, use_int8_skipln, use_int8_multihead, use_qat)
Definition: builder.py:89
bool constructNetwork(SampleUniquePtr< nvinfer1::IBuilder > &builder, SampleUniquePtr< nvinfer1::INetworkDefinition > &network, SampleUniquePtr< nvinfer1::IBuilderConfig > &config)
Uses the API to create the MNIST Network.
Definition: sampleMNISTAPI.cpp:173
unsigned int fused_multihead_attention_v2_int8_384_64_kernel_sm75_cubin_len
Definition: fused_multihead_attention_v2_int8_384_64_kernel.sm75.cpp:18693
float offset
Definition: NvInferPluginUtils.h:100
float visualThreshold
The minimum score threshold to consider a detection.
Definition: sampleUffSSD.cpp:56
int keepTopK
Definition: maskRCNNKernels.h:86
nvinfer1::Weights allocateWeights(int64_t elems, std::uniform_real_distribution< float > distribution=std::uniform_real_distribution< float >(-0.01f, 0.01F))
Definition: caffeWeightFactory.cpp:199
embedding
Definition: demo/Tacotron2/tacotron2/model.py:593
Definition: quant_instancenorm_test.py:46
static half nanh()
Get NaN.
Definition: samples/common/half.h:1759
static trtcaffe::Type getBlobProtoDataType(const trtcaffe::BlobProto &blobMsg)
Definition: caffeWeightFactory.cpp:251
std::vector< float > resizeWeights(int rows, int cols, int rowsNew, int colsNew, const float *memory)
Definition: trtUtil.cpp:41
Definition: sampleReformatFreeIO.cpp:133
size_t getWorkspaceSize(int) const override
Definition: multilevelCropAndResizePlugin.cpp:120
A Softmax layer in a network definition.
Definition: NvInfer.h:2193
float * anchorsDev
Definition: nvFasterRCNNPlugin.h:94
std::vector< float > _h_bias
Definition: instanceNormalizationPlugin.h:93
def dim_param_model()
Definition: onnx_models.py:103
__host__ __device__ __forceinline__ int div(int x) const
Definition: reducedMathPlugin.h:69
std::map< std::string, nvinfer1::ITensor * > mMap
Definition: blobNameToTensor.h:67
~CoordConvACPlugin() override=default
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: reorgPlugin.cpp:88
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: normalizePlugin.cpp:245
void * getDeviceBuffer(const std::string &tensorName) const
Returns the device buffer corresponding to tensorName.
Definition: buffers.h:295
writer of sequences of data
Definition: dataWriter.h:33
const char * SEQ_LEN_IN_BLOB_NAME
Definition: sampleCharRNN.cpp:93
int mNumber
The number to classify.
Definition: sampleMLP.cpp:87
def backward(ctx, grad_outputs, grad_scale)
Definition: tensor_quant.py:281
int32_t w() const
Get the width.
Definition: NvInfer.h:154
def __init__(self, kernel_size, stride=None, padding=0, dilation=1, return_indices=False, ceil_mode=False, **kwargs)
Definition: quant_pooling.py:64
@ kKERNEL
kernel for IConvolutionLayer, IDeconvolutionLayer, or IFullyConnectedLayer
friend struct detail::functions
Definition: ieee_half.h:1086
def test_full_range(self)
Definition: tensor_quant_test.py:125
@ kSOFTSIGN
Softsign activation: x / (1+|x|)
def test_basic(self)
Definition: tf/test_runner.py:33
old_severity
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:44
Definition: quant_conv.py:178
virtual int32_t getVersion() const
Definition: NvUffParser.h:132
def quantize_by_range_fused(x_tuple, num_bits)
Definition: tools/pytorch-quantization/tests/utils.py:32
~GenericBuffer()
Definition: buffers.h:166
static const char * SKIP_LAYER_NORM_INTERLEAVED_NAME
Definition: skipLayerNormInt8InterleavedPlugin.cpp:42
Definition: NvInfer.h:4718
float mFGThreshold
Definition: multilevelProposeROIPlugin.h:141
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: embLayerNormVarSeqlenPlugin.cpp:347
void dumpOutputs(const nvinfer1::IExecutionContext &context, const Bindings &bindings, std::ostream &os)
Print output tensors to stream.
Definition: sampleReporting.cpp:329
forward_transform
Definition: stft.py:50
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: pyramidROIAlignPlugin.cpp:53
def inference(features, tokens)
Definition: inference_varseqlen.py:143
T type
Definition: ieee_half.h:378
context
Definition: backend/trt_legacy.py:311
int mNbWeights
Definition: normalizePlugin.h:97
DetectionOutputParameters params
Definition: nmsPlugin.h:115
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: priorBoxPlugin.cpp:314
int getEndSequenceId() override
Definition: vocabulary.cpp:98
void configurePlugin(const nvinfer1::DynamicPluginTensorDesc *in, int nbInputs, const nvinfer1::DynamicPluginTensorDesc *out, int nbOutputs) override
Definition: skipLayerNormInt8InterleavedPlugin.cpp:129
bool strict
Definition: sampleMovieLens.cpp:528
def test_fake_quant_per_tensor(self)
Definition: quant_instancenorm_test.py:166
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: pyramidROIAlignPlugin.cpp:286
int mTensorVolume
Definition: debugUtil.h:70
void setInput(int32_t index, ITensor &tensor)=0
Append or replace an input of this layer with a specific tensor.
def test_remove(self, field_names)
Definition: test_ir.py:253
std::vector< std::string > splitToStringVec(const std::string &option, char separator)
Definition: sampleOptions.cpp:36
@ kGPU_FALLBACK
Enable layers marked to execute on GPU if layer cannot execute on DLA.
int32_t batch
Definition: argsParser.h:85
@ kDEFAULT
Fail with error when the coordinates are out of bounds. This is the default.
Definition: modules/quant_rnn.py:290
virtual bool supportsOperator(const char *op_name) const =0
Returns whether the specified operator may be supported by the parser.
void copyOutputToHostAsync(const cudaStream_t &stream=0)
Copy the contents of output device buffers to output host buffers asynchronously.
Definition: buffers.h:419
uint16 int2half_impl(T value)
Convert integer to half-precision floating point.
Definition: ieee_half.h:692
bool getBatch(void *bindings[], const char *names[], int nbBindings)
Definition: EntropyCalibrator.h:54
~SLPProjection() override=default
bool operator()(void **ptr, size_t size) const
Definition: buffers.h:200
context
Definition: backend/trt/runner.py:86
def half_function(fn)
Definition: amp_wrapper.py:24
Definition: instanceNormalizationPlugin.h:102
Definition: print_test.py:33
input_metadata
Definition: backend/pyt/runner.py:39
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: nvFasterRCNNPlugin.cpp:148
void setErrorRecorder(nvinfer1::IErrorRecorder *recorder) override
Set the ErrorRecorder for this interface.
Definition: caffeParser.h:59
Structure to define the dimensions of a tensor.
Definition: NvInferRuntimeCommon.h:194
def bert_model(config, init_dict, network, input_tensor, mask_idx, cu_seqlens, max_seqlen)
Definition: builder_varseqlen.py:295
Definition: test_ir.py:221
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: regionPlugin.cpp:381
unsigned int mKernelMetaCount
Definition: fused_multihead_attention.h:279
nvonnxparser::IOnnxConfig::Verbosity mVerbosity
Definition: parserOnnxConfig.h:48
const char * getPluginType() const override
Return the plugin type.
Definition: specialSlicePlugin.cpp:77
def test_fake_quant_per_channel_bias(self)
Definition: quant_conv_transposed_test.py:476
Definition: samples/common/common.h:306
size_t mInputWidth
Definition: proposalPlugin.h:110
virtual const PluginFieldCollection * getFieldNames()=0
Return a list of fields that needs to be passed to createPlugin.
ResizeNearestPluginCreator()
Definition: resizeNearestPlugin.cpp:39
const IBlobNameToTensor * parseBuffers(const char *deployBuffer, size_t deployLength, const char *modelBuffer, size_t modelLength, nvinfer1::INetworkDefinition &network, nvinfer1::DataType weightType) override
Definition: caffeParser.cpp:311
void serialize(void *buffer) const override
Serialize the layer.
Definition: regionPlugin.cpp:304
void setDummyInt8Scales(const IBuilderConfig *c, INetworkDefinition *n)
Definition: samples/common/common.h:563
int mMaxBatchSize
Definition: proposalLayerPlugin.h:102
def squad_output(prefix, config, init_dict, network, input_tensor)
Definition: builder.py:338
def test_validate_nan(self)
Definition: test_comparator.py:133
enable< bool, T, U >::type operator<=(T x, U y)
Comparison for less equal.
Definition: ieee_half.h:2155
int getBatchSize() const override
Definition: BatchStream.h:88
~CoordConvACPluginCreator() override=default
cublasHandle_t mCublas
Definition: normalizePlugin.h:94
void convertAndCopyToDevice(void *&deviceWeights, const nvinfer1::Weights &weights)
Definition: samples/opensource/samplePlugin/fcPlugin.h:251
def get_input_metadata(graph)
Definition: onnx/util.py:157
const char * getPluginVersion() const override
Return the plugin version.
Definition: specialSlicePlugin.cpp:82
bool verbose
Definition: sampleINT8API.cpp:649
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: gridAnchorPlugin.cpp:212
def test_calibrator_metadata_set(self, identity_builder_network)
Definition: trt/test_loader.py:225
def filterData(data, allFeatures, selection)
Definition: prn_utils.py:82
__dict__
Definition: tensor_quant.py:172
def __init__(self, *args, **kwargs)
Definition: modules/quant_rnn.py:295
__half __float2half(float f)
Definition: fp16.h:32
int warmup
Definition: sampleOptions.h:150
nvinfer1::ILayer * parseLRN(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &, BlobNameToTensor &tensors)
Definition: parseLRN.cpp:23
Definition: demo/Tacotron2/tacotron2/model.py:1
virtual int sync()
Definition: samples/common/logging.h:66
T type
Definition: ieee_half.h:373
int initialize() override
Initialize the layer for execution.
Definition: batchTilePlugin.cpp:81
uint16 round_half_up(uint16 value)
Round half-precision number to nearest integer value using round-to-nearest-away-from-zero.
Definition: ieee_half.h:1046
std::vector< int > mMaxOutputSequenceLengths
Definition: beamSearchPolicy.h:73
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleMNISTAPI.cpp:62
def __init__(self, in_channels, out_channels, kernel_size=1, stride=1, padding=None, dilation=1, bias=True, w_init_gain='linear')
Definition: layers.py:37
const char * getPluginType() const override
Return the plugin type.
Definition: cropAndResizePlugin.cpp:98
std::vector< float > mAnchorRatios
Definition: proposalPlugin.h:121
name
Definition: test_misc.py:36
Definition: trt/test_loader.py:171
int mNumOutputs
Definition: slpEmbedder.h:54
Definition: sampleOptions.h:94
NormalizePluginCreator()
Definition: normalizePlugin.cpp:255
bool enable_i2f_trick
Definition: fused_multihead_attention_v2.h:62
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: debugUtil.cpp:86
@ kTRANSPOSE
Like kNONE, but transpose the matrix dimensions.
bool builtin_signbit(T arg)
Check sign.
Definition: ieee_half.h:436
nvinfer1::ILayer * parseReshape(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &, BlobNameToTensor &tensors)
Definition: parseReshape.cpp:23
constexpr int32_t EnumMax< ProfilingVerbosity >()
Maximum number of profile verbosity levels in ProfilingVerbosity enum.
Definition: NvInfer.h:6424
CUresult cuLinkDestroy(CUlinkState state) const
Definition: cudaDriverWrapper.cpp:91
int _y_stride
Definition: split.h:43
Optimization profile for dynamic input dimensions and shape tensors.
Definition: NvInferRuntime.h:1034
void * o_ptr
Definition: fused_multihead_attention_v2.h:32
_name
Definition: tensor_quant.py:82
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:1
@ kREFIT
Enable building a refittable engine.
size_t getDataTypeSize() const
Definition: caffeWeightFactory.cpp:73
static const float DETECTION_MIN_CONFIDENCE
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:44
DataType mType
Definition: samples/common/common.h:331
std::string mName
Definition: pluginLogging.h:303
void print(std::ostream &os) const
Definition: sampleReporting.cpp:265
virtual int node() const =0
index of the ONNX model node in which the error occurred
def __exit__(self, exc_type, exc_value, traceback)
Definition: Polygraphy/polygraphy/util/misc.py:452
FlattenConcatPluginCreator()
Definition: flattenConcat.cpp:292
virtual void setType(DataType type)=0
Set the data type of a tensor.
void writeCalibrationCache(const void *cache, size_t length)
Definition: EntropyCalibrator.h:80
def test_raises(self)
Definition: calibrator_test.py:68
static PluginFieldCollection mFC
Definition: coordConvACPlugin.h:110
def try_receive_on_queue(queue, timeout=None)
Definition: Polygraphy/polygraphy/util/misc.py:253
bool parseHelp(Arguments &arguments)
Definition: sampleOptions.cpp:725
fold_constant
Definition: backend/onnx/loader.py:69
@ DATA_TYPE_INT32
Definition: fused_multihead_attention_common.h:31
num_hidden_layers
Definition: builder.py:95
f
Definition: BERT/inference.py:92
def add(self, name, min, opt, max)
Definition: backend/trt/loader.py:266
ICudaEngine * mEngine
Definition: sampleMovieLensMPS.cpp:245
~RPROIPlugin() override
Definition: nvFasterRCNNPlugin.cpp:98
MultilevelCropAndResizePluginCreator()
Definition: multilevelCropAndResizePlugin.cpp:38
std::vector< std::unique_ptr< ManagedBuffer > > mManagedBuffers
The vector of pointers to managed buffers.
Definition: buffers.h:457
def reset_parameters(self)
Definition: modules/quant_rnn.py:441
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: priorBoxPlugin.cpp:150
SampleINT8(const SampleINT8Params ¶ms)
Definition: sampleINT8.cpp:64
def test_overflow_fp16(self)
Definition: tensor_quant_test.py:110
def test_regex(self, resnet18)
Definition: optim_helper_test.py:43
expr remainder(half x, half y)
Remainder of division.
Definition: ieee_half.h:2253
int iterations
Definition: demo/BERT/infer_c/common.h:32
virtual bool getPrintLayerInfo() const =0
Get whether the layer information will be printed.
std::shared_ptr< CudaBind< float > > mRegWeightDevice
Definition: multilevelProposeROIPlugin.h:111
static T cast_impl(half arg, true_type)
Definition: samples/common/half.h:2650
~PriorBox() override=default
std::ostream * mOstream
Definition: demo/BERT/infer_c/logging.h:74
static void reportTestResult(const TestAtom &testAtom, TestResult result)
method that implements logging test results
Definition: samples/common/logging.h:418
Definition: comparator/test_struct.py:27
static expr tan(float arg)
Tan implementation.
Definition: samples/common/half.h:1902
const errorPair & operator[](size_t index) const noexcept
Definition: ErrorRecorder.h:112
def test_calibration(self)
Definition: integration_test.py:145
~TrtCudaGraph()
Definition: sampleDevice.h:178
constexpr const char * GROUP_NORM_VERSION
Definition: groupNormalizationPlugin.cpp:37
def get_data_loader(model_name, dataset_path, audiopaths_and_text, args)
Definition: data_functions.py:37
virtual void finalize()=0
it is called right after inference ends
BatchedNMSDynamicPlugin(NMSParameters param)
Definition: batchedNMSPlugin.cpp:58
RegionParameters params
Definition: regionPlugin.h:121
def topk_func(k=10, axis=-1, outputs=None, exclude=None)
Definition: postprocess.py:27
@ kHALF
IEEE 16-bit floating-point format.
SampleConfig()
Definition: sampleConfig.h:62
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: specialSlicePlugin.cpp:99
float mNegSlope
Definition: lReluPlugin.h:72
std::string mNamespace
Definition: embLayerNormVarSeqlenPlugin.h:86
unsigned char fused_multihead_attention_v2_fp16_256_64_kernel_sm75_cubin[]
Definition: fused_multihead_attention_v2_fp16_256_64_kernel.sm75.cpp:19
def __getitem__(self, index)
Definition: comparator/data_loader.py:29
bool supportsFormatCombination(int pos, const nvinfer1::PluginTensorDesc *inOut, int nbInputs, int nbOutputs) override
Definition: qkvToContextInt8InterleavedPlugin.cpp:118
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: generateDetectionPlugin.cpp:316
expr tgamma(half arg)
Gamma function.
Definition: ieee_half.h:2540
virtual void setProfiler(IProfiler *) noexcept=0
Set the profiler.
def test_o_multiple_outputs(self)
Definition: test_ir.py:100
A RaggedSoftmax layer in a network definition.
Definition: NvInfer.h:4241
unsigned int fused_multihead_attention_v2_fp16_96_64_kernel_sm80_cubin_len
Definition: fused_multihead_attention_v2_fp16_96_64_kernel.sm80.cpp:2788
~Region() override=default
half(detail::expr rhs)
Copy constructor.
Definition: samples/common/half.h:1408
const std::vector< char > idToChar
Definition: sampleCharRNN.cpp:108
def test_no_quant(self, verbose)
Definition: quant_rnn_test.py:283
def transform(self, input_data)
Definition: stft.py:77
virtual bool isExecutionTensor() const =0
Whether the tensor is an execution tensor.
def forward(self, memory, decoder_inputs, memory_lengths)
Definition: demo/Tacotron2/tacotron2/model.py:439
def relu(self, a)
Definition: 07_creating_a_model_with_the_layer_api/generate.py:59
_entries
Definition: cmudict.py:44
def __repr__(self)
Definition: tensor.py:136
std::string removeSingleQuotationMarks(std::string &str)
Definition: sampleOptions.cpp:229
SampleMovieLensParams mParams
Definition: sampleMovieLens.cpp:149
enable< bool, T, U >::type operator>=(T x, U y)
Comparison for greater equal.
Definition: ieee_half.h:2162
def __init__(self, dataset_path, audiopaths_and_text, args)
Definition: waveglow/data_function.py:29
location_layer
Definition: demo/Tacotron2/tacotron2/model.py:56
def __init__(self, initial_tensors=None)
Definition: graph.py:30
Definition: tools/onnx-graphsurgeon/onnx_graphsurgeon/util/__init__.py:1
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: batchTilePlugin.cpp:137
_calib_hist
Definition: histogram.py:50
Definition: comparator/struct.py:22
@ kFAILED
The test failed.
unsigned int fused_multihead_attention_v2_int8_192_64_kernel_sm80_cubin_len
Definition: fused_multihead_attention_v2_int8_192_64_kernel.sm80.cpp:15004
Definition: tools/Polygraphy/polygraphy/backend/trt/__init__.py:1
void reset(int firstBatch) override
Definition: BatchStream.h:208
static bool isfinite(half arg)
Classification implementation.
Definition: samples/common/half.h:2342
int getBatchSize() const
Definition: frcnnUtils.h:314
def test_weight_fake_quant_per_channel(self)
Definition: quant_conv_transposed_test.py:374
virtual void detachFromContext()
Detach the plugin object from its execution context.
Definition: NvInferRuntimeCommon.h:675
Weights * mDeviceWidths
Definition: gridAnchorPlugin.h:98
Definition: sampleCharRNN.cpp:248
virtual ~ITopKLayer()
Definition: NvInfer.h:4123
IErrorRecorder::RefCount incRefCount() noexcept final
Increments the refcount for the current ErrorRecorder.
Definition: ErrorRecorder.h:101
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: multilevelProposeROIPlugin.cpp:328
@ kINFO
Informational messages with instructional information.
def __str__(self)
Definition: cuda.py:199
bool graph
Definition: sampleOptions.h:158
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: nvFasterRCNNPlugin.cpp:281
bool kept
Definition: bboxUtils.h:45
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleCharRNN.cpp:220
int mSampleCount
Definition: benchmarkWriter.h:48
std::vector< float > mCandidateLikelihoods
Definition: beamSearchPolicy.h:77
half & operator=(float rhs)
Assignment operator.
Definition: samples/common/half.h:1479
static half_float::half signaling_NaN()
Signalling NaN.
Definition: samples/common/half.h:4258
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: flattenConcat.cpp:126
LogStreamConsumer(Severity reportableSeverity, Severity severity)
Creates a LogStreamConsumer which logs messages with level severity.
Definition: pluginLogging.h:134
outputs
Definition: backend/onnx/loader.py:125
bool parseArgs(Args &args, int32_t argc, char *argv[])
Populates the Args struct with the provided command-line parameters.
Definition: argsParser.h:99
virtual ISliceLayer * addSlice(ITensor &input, Dims start, Dims size, Dims stride)=0
Add a slice layer to the network.
def infer_onnx(self, spect, z, sigma=0.9)
Definition: export_waveglow_onnx.py:109
def get_batch_size(self)
Definition: demo/BERT/helpers/calibrator.py:52
void maskPPM(PPM< uint8_t > &image, const PPM< uint8_t > &mask, const int start_x, const int start_y, const std::vector< int > &color)
Definition: sampleUffMaskRCNN.cpp:230
@ kV2_DYNAMICEXT
IPluginV2IOExt.
float outStart
Definition: sampleReporting.h:98
def test_export_constant_tensor_to_value_info_proto(self)
Definition: test_exporters.py:47
Definition: modules/quant_rnn.py:381
int getOutputSize() override
get the size of raw logits vector
Definition: slpProjection.cpp:59
Definition: quant_conv.py:90
tacotron2
Definition: export_tacotron2_onnx.py:79
const FieldMap * fields
Definition: NvUffParser.h:90
std::istream & operator>>(std::istream &input, Vocabulary &value)
Definition: vocabulary.cpp:61
void * mPtr
Definition: maskRCNNKernels.h:66
profiles
Definition: backend/trt/loader.py:325
void serialize(void *buffer) const override
Serialize the layer.
Definition: batchedNMSPlugin.cpp:272
_track_amax
Definition: max.py:39
def set_output_range(layer, maxval, out_idx=0)
Definition: builder.py:112
Definition: tokenization.py:330
def mark_layerwise(model)
Definition: onnx/util.py:91
def test_fake_quant_per_tensor_with_bias(self)
Definition: quant_linear_test.py:120
std::string getInfo() override
get the textual description of the component
Definition: contextNMT.cpp:40
def test_fake_quant_per_channel_other_prec(self)
Definition: quant_conv_transposed_test.py:163
int read(int samplesToRead, int maxInputSequenceLength, int *hInputData, int *hActualInputSequenceLengths) override
reads the batch of smaples/sequences
Definition: limitedSamplesDataReader.cpp:31
std::vector< std::string > classify(const std::vector< std::string > &refVector, const std::vector< T > &output, const size_t topK)
Definition: samples/common/common.h:428
unsigned char fused_multihead_attention_v2_int8_256_64_kernel_cubin[]
Definition: fused_multihead_attention_v2_int8_256_64_kernel.sm72.cpp:19
parser
Definition: evaluate-v1.1.py:99
int H
Definition: normalizePlugin.h:102
std::string mNameSpace
Definition: generateDetectionPlugin.h:109
def parse_tacotron2_args(parent, add_help=False)
Definition: tacotron2/arg_parser.py:22
TrtUniquePtr< nvuffparser::IUffParser > uffParser
Definition: sampleEngines.h:36
The SampleSSD class implements the SSD sample.
Definition: sampleSSD.cpp:60
def copy(self, List["Tensor"] inputs=None, List["Tensor"] outputs=None)
Definition: node.py:96
std::unordered_map< std::string, int > mNames
Definition: sampleUtils.h:451
const char * getPluginType() const override
Return the plugin type.
Definition: batchedNMSPlugin.cpp:360
unsigned char fused_multihead_attention_fp16_128_64_kernel_sm75_cu_o[]
Definition: fused_multihead_attention_fp16_128_64_kernel.sm75.cpp:20
@ kAND
Logical AND of two elements.
Class used for parsing Caffe models.
Definition: NvCaffeParser.h:169
T type
Definition: ieee_half.h:383
def test_learn_amax(self)
Definition: tensor_quantizer_test.py:93
bool threads
Definition: sampleOptions.h:157
void destroyPlugin()
Definition: samples/opensource/samplePlugin/fcPlugin.h:376
SampleUniquePtr< nvinfer1::ICudaEngine > mPredictionEngine
Definition: sampleDynamicReshape.cpp:84
Definition: test_misc.py:34
def __init__(self)
Definition: cuda.py:33
BatchTilePluginCreator()
Definition: batchTilePlugin.cpp:183
static long lrint(half arg)
Nearest integer implementation.
Definition: samples/common/half.h:2126
@ kINT8
Enable Int8 layer selection, with FP32 fallback with FP16 fallback if kFP16 also specified.
virtual void parse(Arguments &arguments)=0
def get_trt_dtype(self)
Definition: builder_varseqlen.py:103
def __call__(self)
Definition: backend/tf/loader.py:150
static PluginFieldCollection mFC
Definition: nvFasterRCNNPlugin.h:121
Definition: classification_flow_test.py:27
def __init__(self, tf_loader, save_uff=None, preprocessor=None)
Definition: backend/trt_legacy.py:49
ids_to_tokens
Definition: tokenization.py:189
Severity mSeverity
Definition: samples/common/logging.h:179
unsigned long type
Definition: ieee_half.h:341
def __call__(self)
Definition: backend/trt_legacy.py:41
~CropAndResizePlugin() override
Definition: cropAndResizePlugin.cpp:96
def __repr__(self)
Definition: max.py:99
void operator()(T *t)
Definition: sampleUtils.h:459
def convert_tokens_to_ids(self, tokens)
Definition: tokenization.py:177
def test_extend_2_args_2_rv(self)
Definition: test_func.py:92
expr atanh(half arg)
Hyperbolic area tangent.
Definition: ieee_half.h:2508
The SampleAlgorithmSelector class implements the SampleAlgorithmSelector sample.
Definition: sampleAlgorithmSelector.cpp:312
std::string mCacheFileName
Definition: sampleAlgorithmSelector.cpp:114
~UffBufferShutter()
Definition: sampleEngines.cpp:53
half(float rhs)
Conversion constructor.
Definition: samples/common/half.h:1415
nvinfer1::Dims getOutputDimensions(int index, const nvinfer1::Dims *inputs, int nbInputDims) override
Definition: samples/opensource/samplePlugin/fcPlugin.h:100
~GpuTimer()
Definition: samples/common/common.h:816
def test_fp16(self, identity_builder_network, flag)
Definition: trt/test_loader.py:210
Definition: lReluPlugin.h:76
float mIOUThreshold
Definition: generateDetectionPlugin.h:98
std::map< std::string, nvinfer1::Weights > mWeightMap
The weight name to weight value map.
Definition: sampleMNISTAPI.cpp:91
bool build()
Builds the network engine.
Definition: sampleMNIST.cpp:112
~MultilevelProposeROIPluginCreator()
Definition: multilevelProposeROIPlugin.h:125
virtual void setGpuAllocator(IGpuAllocator *allocator) noexcept=0
Set the GPU allocator.
def save_checkpoint(model, optimizer, epoch, config, amp_run, output_dir, model_name, local_rank, world_size)
Definition: train.py:173
const char * name
Definition: NvUffParser.h:79
unsigned char fused_multihead_attention_int8_384_64_kernel_sm75_cu_o[]
Definition: fused_multihead_attention_int8_384_64_kernel.sm75.cpp:20
virtual int32_t getVersion() const
Definition: NvCaffeParser.h:119
Definition: test_cuda.py:39
data
Definition: demo/BERT/helpers/calibrator.py:36
Definition: tools/pytorch-quantization/tests/fixtures/models.py:26
half ldexp(half arg, int exp)
Multiply by power of two.
Definition: ieee_half.h:2635
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: generateDetectionPlugin.cpp:66
def attention_layer_opt(prefix, config, init_dict, network, input_tensor, imask)
Definition: builder.py:124
int32_t userId
Definition: sampleMovieLensMPS.cpp:157
cudnnTensorDescriptor_t bnDesc
Definition: groupNormalizationPlugin.h:103
size_t mInputHeight
Definition: proposalPlugin.h:109
def restore_float_modules(self)
Definition: quant_modules.py:102
IPluginV2Ext * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: nmsPlugin.cpp:383
def reshape_network()
Definition: trt/test_loader.py:58
A divUp(A x, B n)
Definition: samples/common/common.h:646
static bool islessequal(half x, half y)
Comparison implementation.
Definition: samples/common/half.h:2444
def test_weight_fake_quant_per_channel(self)
Definition: quant_conv_test.py:90
IPluginV2DynamicExt * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: batchedNMSPlugin.cpp:636
bool build()
Function builds the network engine.
Definition: sampleUffSSD.cpp:125
def identity(self, inp)
Definition: 08_replacing_a_subgraph/generate.py:31
const char * getPluginVersion() const override
Return the plugin version.
Definition: proposalLayerPlugin.cpp:56
int oW
Definition: coordConvACPlugin.h:87
def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros', **kwargs)
Definition: quant_conv.py:139
weight_ih
Definition: modules/quant_rnn.py:346
const char * getPluginVersion() const override
Return the plugin version.
Definition: qkvToContextInt8InterleavedPlugin.cpp:172
std::vector< LayerProfile >::iterator mIterator
Definition: sampleReporting.h:189
GenericBuffer< DeviceAllocator, DeviceFree > DeviceBuffer
Definition: buffers.h:216
constexpr int32_t EnumMax< QuantizationFlag >()
Maximum number of quantization flags in QuantizationFlag enum.
Definition: NvInfer.h:6364
Definition: optim_helper_test.py:95
MultiplicativeAlignment(ComponentWeights::ptr weights)
Definition: multiplicativeAlignment.cpp:24
def default_value(value, default)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:49
bool teardown()
Used to clean up any state created in the sample class.
Definition: sampleMovieLens.cpp:448
Vocabulary()
Definition: vocabulary.cpp:29
sampling_rate
Definition: waveglow/data_function.py:32
def test_no_quant(self)
Definition: quant_instancenorm_test.py:99
def scan_model()
Definition: onnx_models.py:142
def __repr__(self)
Definition: tensor.py:229
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: detectionLayerPlugin.cpp:118
Definition: test_ir.py:134
virtual ILoop * addLoop() noexcept=0
Add a loop to the network.
def collapse_whitespace(text)
Definition: cleaners.py:75
int initialize() override
Initialize the layer for execution.
Definition: proposalLayerPlugin.cpp:130
Definition: postprocess.py:1
"onnx.ModelProto" export_onnx(Graph graph, do_type_check=True, **kwargs)
Definition: onnx_exporter.py:93
_fake_quant
Definition: tensor_quant.py:84
int initialize() override
Initialize the layer for execution.
Definition: lReluPlugin.cpp:98
def test_entropy_and_percentile_calib(self)
Definition: tensor_quantizer_test.py:217
SampleMNIST(const samplesCommon::CaffeSampleParams ¶ms)
Definition: sampleMNIST.cpp:54
virtual ITensor * getOutput(int32_t index) const =0
Get the layer output corresponding to the given index.
Definition: textWriter.h:33
std::unordered_map< std::string, std::vector< int > > shapes
Definition: sampleOptions.h:162
void serialize(void *buffer) const override
Serialize the layer.
Definition: proposalLayerPlugin.cpp:197
def test_can_convert_in_place_to_constant(self)
Definition: test_ir.py:31
def __setitem__(self, key, value)
Definition: comparator/struct.py:110
Definition: quant_pooling_test.py:59
bool isNormalized
Definition: NvInferPluginUtils.h:252
void setTolerance(float t)
Definition: sampleConfig.h:301
constexpr int32_t EnumMax< CalibrationAlgoType >()
Maximum number of elements in CalibrationAlgoType enum.
Definition: NvInfer.h:5982
nvinfer1::ILayer * addLSTMLayers(SampleCharRNNBase::SampleUniquePtr< nvinfer1::INetworkDefinition > &network) final
Add inputs to the TensorRT network and configure LSTM layers using network definition API.
Definition: sampleCharRNN.cpp:566
AlgorithmCacheReader(const std::string &cacheFileName)
Definition: sampleAlgorithmSelector.cpp:191
std::string mPluginNamespace
Definition: priorBoxPlugin.h:95
bool hasOverflowed() const noexcept final
Determine if the error stack has overflowed.
Definition: ErrorRecorder.h:59
def test_backward(self)
Definition: tensor_quant_test.py:178
int mNumPriors
Definition: priorBoxPlugin.h:93
virtual int32_t getTensorRTVersion() const
Return the API version with which this plugin was built.
Definition: NvInferRuntimeCommon.h:409
def get_model(model_name, model_config, to_cuda, uniform_initialize_bn_weight=False, forward_is_infer=False)
Definition: demo/Tacotron2/models.py:54
_param_buf_size
Definition: modules/quant_rnn.py:139
def read_squad_json(input_file)
Definition: data_processing.py:186
int inputW
The input width.
Definition: sampleMLP.cpp:48
def tokenize(self, text)
Definition: tokenization.py:338
samplesCommon::DeviceBuffer mPredictionInput
Device buffer for the output of the preprocessor, i.e.
Definition: sampleDynamicReshape.cpp:89
virtual bool getBroadcastAcrossBatch() const =0
Check if tensor is broadcast across the batch.
static std::basic_istream< charT, traits > & read(std::basic_istream< charT, traits > &in, half &arg)
Input implementation.
Definition: ieee_half.h:1251
Definition: sampleCharRNN.cpp:223
~Semaphore()
Definition: sampleMovieLensMPS.cpp:88
virtual ~IShuffleLayer()
Definition: NvInfer.h:3849
static std::vector< PluginField > mPluginAttributes
Definition: multilevelProposeROIPlugin.h:143
Weights deserializeToDevice(const char *&hostBuffer, size_t count)
Definition: normalizePlugin.cpp:158
name
Definition: backend/tf/loader.py:176
const std::string mLayerName
Definition: qkvToContextInt8InterleavedPlugin.h:78
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: embLayerNormVarSeqlenPlugin.cpp:412
def test_optimizer_feed(self, resnet18)
Definition: optim_helper_test.py:71
def version(version_str)
Definition: Polygraphy/polygraphy/util/misc.py:32
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: specialSlicePlugin.cpp:206
def test_import_graph_with_initializer(self)
Definition: test_importers.py:113
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: reorgPlugin.cpp:132
nvinfer1::DimsHW argMaxLabelDims
Definition: maskRCNNKernels.h:131
TRTException(const char *fl, const char *fn, int ln, int st, const char *msg, const char *nm)
Definition: checkMacrosPlugin.h:77
std::vector< size_t > mMemSizes
Definition: sampleMovieLensMPS.cpp:250
static const Dims3 MODEL_INPUT_SHAPE
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:172
Definition: backend/trt/loader.py:385
bool isNormalized
Definition: NvInferPluginUtils.h:189
void serializeFromDevice(char *&hostBuffer, Weights deviceWeights) const
Definition: normalizePlugin.cpp:152
key
Definition: quant_bert.py:65
virtual void serialize(void *buffer)=0
Serialize the layer.
static expr cbrt(float arg)
Cubic root implementation.
Definition: samples/common/half.h:1847
void parse(Arguments &arguments) override
Definition: sampleOptions.cpp:650
virtual int32_t getTensorsWithDynamicRange(int32_t size, const char **tensorNames) const =0
Get names of all tensors that have refittable dynamic ranges.
Definition: tools/pytorch-quantization/tests/fixtures/models.py:44
int max
Definition: samples/common/common.h:663
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: specialSlicePlugin.cpp:104
std::shared_ptr< Embedder > ptr
Definition: embedder.h:35
float y2
Definition: sampleUffMaskRCNN.cpp:54
dtype
Definition: cuda.py:123
void copyInputToDevice()
Copy the contents of input host buffers to input device buffers synchronously.
Definition: buffers.h:395
def infer_shapes(model)
Definition: onnx/util.py:40
static const Dims3 MODEL_INPUT_SHAPE
Definition: tlt_mrcnn_config.h:172
def test_raises(self)
Definition: optim_helper_test.py:81
MultilevelCropAndResize(int pooled_size, const nvinfer1::Dims &image_size)
Definition: multilevelCropAndResizePlugin.cpp:89
virtual void setPluginNamespace(const char *pluginNamespace)=0
Set the namespace of the plugin creator based on the plugin library it belongs to.
ResizeMode
Enumerates various modes of resize in the resize layer. Resize mode set using setResizeMode().
Definition: NvInfer.h:4331
std::vector< char > mWeights
Definition: componentWeights.h:42
void * safeCudaMalloc(size_t memSize)
Definition: samples/common/common.h:358
def test_shape_output(self)
Definition: trt/test_runner.py:59
virtual nvonnxparser::IOnnxConfig::Verbosity getVerbosityLevel() const
Get the Verbosity Level.
Definition: parserOnnxConfig.h:95
TensorFormat
Format of the input/output tensors.
Definition: NvInferRuntimeCommon.h:225
quant_conv
Definition: print_test.py:48
nvinfer1::Dims getOutputDimensions(int index, const nvinfer1::Dims *inputs, int nbInputDims) override
Definition: debugUtil.cpp:38
Definition: batchedNMSPlugin.h:76
start
Definition: demo/Tacotron2/waveglow/model.py:101
static std::vector< PluginField > mPluginAttributes
Definition: detectionLayerPlugin.h:131
def test_cuda_ext_with_axis(self)
Definition: tensor_quant_test.py:217
e
Definition: 05_folding_constants/generate.py:33
Definition: quant_conv_test.py:48
def test_no_quant(self)
Definition: quant_conv_transposed_test.py:50
static TestAtom defineTest(const std::string &name, int argc, char const *const *argv)
A convenience overloaded version of defineTest() that accepts an array of command-line arguments as i...
Definition: pluginLogging.h:332
static const int RPN_ANCHOR_STRIDE
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:72
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: groupNormalizationPlugin.cpp:162
static int reportTest(const TestAtom &testAtom, bool pass)
Definition: samples/common/logging.h:365
void dumpBuffer(const void *buffer, int volume, const std::string &separator, std::ostream &os)
Definition: sampleUtils.h:204
Definition: demo/BERT/helpers/calibrator.py:1
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleFasterRCNN.cpp:92
void exportJSONProfile(const std::string &fileName) const
Export a profile to JSON file.
Definition: sampleReporting.cpp:302
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: priorBoxPlugin.cpp:288
@ kWARNING
Application error has been discovered. TensorRT has recovered or fallen back to a default.
unsigned char fused_multihead_attention_v2_int8_128_64_kernel_sm75_cubin[]
Definition: fused_multihead_attention_v2_int8_128_64_kernel.sm75.cpp:19
void reset(void *ptr=nullptr)
Definition: sampleDevice.h:255
Dims3(int32_t d0, int32_t d1, int32_t d2)
Construct a Dims3 from 3 elements.
Definition: NvInfer.h:183
outputs
Definition: tensor.py:199
nodes
Definition: graph.py:114
size_t mRpnHeight
Definition: proposalPlugin.h:111
virtual ~IOnnxConfig()
Definition: NvOnnxConfig.h:48
Definition: backend/trt/loader.py:489
explicit_batch
Definition: backend/trt/loader.py:101
void serialize(void *buffer) const override
Serialize the layer.
Definition: qkvToContextInt8InterleavedPlugin.cpp:195
int initialize() override
Initialize the layer for execution.
Definition: regionPlugin.cpp:376
def activate_impl(self)
Definition: backend/trt_legacy.py:228
inputs
Definition: graph.py:115
attention_layer
Definition: export_tacotron2_onnx.py:121
def test_no_shape_tensor_false_positive_float(self)
Definition: test_data_loader.py:75
SampleUniquePtr< nvcaffeparser1::IBinaryProtoBlob > mMeanBlob
Definition: sampleMNIST.cpp:101
DataType iType
Definition: coordConvACPlugin.h:85
network_loader
Definition: backend/trt_legacy.py:215
def test_i_multiple_inputs(self)
Definition: test_ir.py:196
int duration
Definition: sampleOptions.h:151
static PluginCreatorRegistry & getInstance()
Definition: InferPlugin.cpp:72
const char * getPluginVersion() const override
Return the plugin version.
Definition: resizeNearestPlugin.cpp:170
The SampleMNISTAPI class implements the MNIST API sample.
Definition: sampleMNISTAPI.cpp:59
void destroy()
Destroy IOnnxConfig object.
Definition: sampleConfig.h:315
uint16 float2half(T value)
Convert floating point to half-precision.
Definition: ieee_half.h:681
Definition: maskRCNNKernels.h:155
PoolParameters mPoolingParams
Definition: sampleUffPluginV2Ext.cpp:634
int C2
Definition: nmsPlugin.h:90
int mInputSize
Definition: pyramidROIAlignPlugin.h:98
Definition: generateDetectionPlugin.h:36
float visualizeThreshold
Definition: sampleUffFasterRCNN.cpp:74
const char * getPluginVersion() const override
Return the plugin version.
Definition: resizeNearestPlugin.cpp:52
def test_pop_index(self, field_names)
Definition: test_ir.py:269
SplitPlugin(void const *serialData, size_t serialLength)
Definition: split.h:81
def axis(self)
Definition: tensor_quant.py:120
def test_raise(self)
Definition: quant_linear_test.py:46
@ kDHWC8
Eight channel format where C is padded to a multiple of 8.
virtual __attribute__((deprecated)) virtual nvinfer1 void setMaxBatchSize(int32_t batchSize)=0
Create a network definition object where all tensors have an implicit batch dimension.
std::vector< int32_t > allItems
Definition: sampleMovieLensMPS.cpp:160
def compute_amax(self, *args, **kwargs)
Definition: tools/pytorch-quantization/pytorch_quantization/calib/calibrator.py:48
def __call__(self)
Definition: backend/pyt/loader.py:19
def pad_sequences(batch)
Definition: Tacotron2/inference.py:130
def remove(conv_list)
Definition: demo/Tacotron2/waveglow/model.py:283
def info(self, message, mode=LogMode.EACH)
Definition: Polygraphy/polygraphy/logger/logger.py:266
~BlobNameToTensor() override=default
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: nmsPlugin.cpp:166
explicit_precision
Definition: backend/trt/loader.py:71
def test_warmup_runs(self)
Definition: test_comparator.py:33
std::string getInfo() override
get the textual description of the component
Definition: softmaxLikelihood.cpp:103
@ kRUNNING
The test is running.
def prepare_state(self, float_module_list=None, custom_map=None)
Definition: quant_modules.py:65
def test_equals_op_mismatch(self)
Definition: test_ir.py:147
Definition: sampleMovieLens.cpp:49
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: batchedNMSPlugin.cpp:80
Definition: priorBoxPlugin.h:98
def add(self, a, b)
Definition: 07_creating_a_model_with_the_layer_api/generate.py:34
std::vector< int > nms_classifier(std::vector< float > &boxes_per_cls, std::vector< float > &probs_per_cls, float NMS_OVERLAP_THRESHOLD, int NMS_MAX_BOXES)
NMS helper function in post-processing.
Definition: sampleUffFasterRCNN.cpp:617
samplesCommon::OnnxSampleParams mParams
The parameters for the sample.
Definition: sampleDynamicReshape.cpp:78
Definition: tools/Polygraphy/polygraphy/comparator/__init__.py:1
virtual void log(Severity severity, const char *msg)=0
A callback implemented by the application to handle logging messages;.
const IDimensionExpr * d[Dims::MAX_DIMS]
The extent of each dimension.
Definition: NvInferRuntime.h:379
const char * getPluginVersion() const override
Return the plugin version.
Definition: groupNormalizationPlugin.cpp:72
def test_raise(self)
Definition: quant_pooling_test.py:151
def __init__(self, graph, opset=None, optimize=None, fold_constant=None)
Definition: backend/onnx/loader.py:50
T * mBuffer
Definition: pinnedHostBuffer.h:62
const char * COORDCONV_AC_PLUGIN_VERSION
Definition: coordConvACPlugin.cpp:27
def __len__(self)
Definition: waveglow/data_function.py:68
std::string mEngineFilename
Definition: sampleConfig.h:37
def __init__(self, model_bytes, explicit_precision=None)
Definition: backend/trt/loader.py:112
def main()
Definition: inference_perf.py:51
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: flattenConcat.cpp:82
int gCurrentPosition
Definition: limitedSamplesDataReader.h:43
def disable_calib(self)
Definition: tensor_quantizer.py:172
def unsigned(self)
Definition: tensor_quant.py:144
def checkpoint_from_distributed(state_dict)
Definition: Tacotron2/inference.py:71
void setPluginFactoryExt(nvcaffeparser1::IPluginFactoryExt *factory) override
Set the IPluginFactoryExt used to create the user defined pluginExts.
Definition: caffeParser.h:49
std::string magic
Definition: samples/common/common.h:654
wordpiece_tokenizer
Definition: tokenization.py:167
def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0, output_padding=0, groups=1, bias=True, dilation=1, padding_mode='zeros', **kwargs)
Definition: quant_conv.py:314
def get_predictions(doc_tokens, features, results, n_best_size, max_answer_length)
Definition: data_processing.py:333
def setup_method(self)
Definition: test_api.py:26
SampleMNISTAPIParams mParams
The parameters for the sample.
Definition: sampleMNISTAPI.cpp:87
virtual ITripLimitLayer * addTripLimit(ITensor &tensor, TripLimit limit) noexcept=0
Add a trip-count limiter, based on the given tensor.
def stream_synchronize(self, stream)
Definition: cuda.py:51
int mBatchSize
The batch size for legacy networks, 0 otherwise.
Definition: buffers.h:456
void configurePlugin(const nvinfer1::DynamicPluginTensorDesc *in, int nbInputs, const nvinfer1::DynamicPluginTensorDesc *out, int nbOutputs) override
Definition: groupNormalizationPlugin.cpp:200
def test_fake_quant_per_tensor(self)
Definition: quant_instancenorm_test.py:114
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: pyramidROIAlignPlugin.cpp:164
void fillBufferHalf(void *buffer, int volume, H min, H max)
Definition: sampleUtils.h:185
~SampleConfig()
Definition: sampleConfig.h:73
Definition: sampleUffPluginV2Ext.cpp:188
int w
Definition: samples/common/common.h:663
nvinfer1::Weights mKernelWeights
Definition: slpEmbedder.h:52
virtual int sync()
Definition: pluginLogging.h:62
Definition: backend/trt/loader.py:94
virtual ~IPluginV2DynamicExt()
Definition: NvInferRuntime.h:539
attention_rnn_dim
Definition: demo/Tacotron2/tacotron2/model.py:245
const FusedMultiHeadAttentionXMMAKernelV2 * mXmmaKernel
Definition: qkvToContextInt8InterleavedPlugin.h:88
half ceil(half arg)
Nearest integer not less than half value.
Definition: ieee_half.h:2551
unsigned int fused_multihead_attention_fp16_64_64_kernel_sm75_cu_o_len
Definition: fused_multihead_attention_fp16_64_64_kernel.sm75.cpp:1438
TFusedMHAKernelFactory()=default
use_fp16
Definition: builder.py:97
def test_quant_input_hidden(self, verbose)
Definition: quant_rnn_test.py:165
LogStream< ILogger::Severity::kINFO > gLogInfo
Definition: checkMacrosPlugin.cpp:50
int outputBboxSize
Definition: sampleUffFasterRCNN.cpp:72
d
Definition: 05_folding_constants/generate.py:32
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: reorgPlugin.cpp:173
virtual void initialize()=0
it is called right before inference starts
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: gridAnchorPlugin.cpp:173
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: gridAnchorPlugin.cpp:331
int32_t numAspectRatios
Definition: NvInferPluginUtils.h:147
size_t mSize
Definition: sampleDevice.h:349
SystemOptions system
Definition: sampleOptions.h:190
def __init__(self, name, map, index, expected)
Definition: test_misc.py:35
static TestAtom defineTest(const std::string &name, int argc, char const *const *argv)
A convenience overloaded version of defineTest() that accepts an array of command-line arguments as i...
Definition: samples/common/logging.h:311
std::mutex mLogMutex
Definition: pluginLogging.h:116
void throwCublasError(const char *file, const char *function, int line, int status, const char *msg)
Definition: checkMacrosPlugin.cpp:62
def indent(self, level=1)
Definition: Polygraphy/polygraphy/logger/logger.py:162
void * data() const noexcept override
A pointer to the raw data that is owned by the library.
Definition: samples/common/common.h:310
def test_two_tensor(self, verbose)
Definition: calibrator_test.py:279
LogStreamConsumer LOG_FATAL(const Logger &logger)
produces a LogStreamConsumer object that can be used to log messages of severity kINTERNAL_ERROR
Definition: pluginLogging.h:522
std::ostream & mOutput
Definition: samples/common/logging.h:102
bool empty() const noexcept
Simple helper function that.
Definition: ErrorRecorder.h:80
TRTParsedArgs getOptions(int argc, const char *const *argv, const std::vector< TRTOption > &options)
Parse the input arguments passed to main() and extract options as well as positional arguments.
Definition: getOptions.cpp:237
nvinfer1::Dims mInputDims
The dimensions of the input to the network.
Definition: sampleOnnxMNIST.cpp:71
xy_t mPooledSize
Definition: pyramidROIAlignPlugin.h:94
def test_simple_run(self)
Definition: tensor_quant_test.py:137
nvinfer1::DimsNCHW mDimensions
Definition: binaryProtoBlob.h:63
SampleBuffer & operator=(SampleBuffer &&sampleBuffer) noexcept
Definition: sampleReformatFreeIO.cpp:160
virtual void resetDynamicRange()=0
Undo effect of setDynamicRange.
def __init__(self, *args, **kwargs)
Definition: modules/quant_rnn.py:303
deploy
Definition: backend/trt_legacy.py:145
The SamplePlugin class implements samplePlugin.
Definition: samplePlugin.cpp:55
def test_print_calibrator(self)
Definition: print_test.py:55
name_idx
Definition: graph.py:126
int initialize() override
Initialize the layer for execution.
Definition: multilevelCropAndResizePlugin.cpp:106
static T cast(half arg)
Definition: samples/common/half.h:2644
CUresult cuLinkComplete(CUlinkState state, void **cubinOut, size_t *sizeOut) const
Definition: cudaDriverWrapper.cpp:81
IErrorRecorder::ErrorDesc getErrorDesc(int32_t errorIdx) const noexcept final
Returns the c-style string description of the error.
Definition: ErrorRecorder.h:54
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: multilevelCropAndResizePlugin.cpp:57
n_group
Definition: demo/Tacotron2/waveglow/model.py:166
float expectedPredictedMaxRatingItemProb
Definition: sampleMovieLensMPS.cpp:159
static PluginFieldCollection mFC
Definition: priorBoxPlugin.h:116
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: batchTilePlugin.cpp:97
void deserializeToDevice(const char *&hostBuffer, void *&deviceWeights, size_t size)
Definition: samples/opensource/samplePlugin/fcPlugin.h:302
T read(const char *&buffer) const
Definition: sampleUffPluginV2Ext.cpp:587
SplitPlugin(int axis, int *const &output_lengths, int noutput)
Definition: split.h:69
An Activation layer in a network definition.
Definition: NvInfer.h:1643
T type
Definition: ieee_half.h:376
int32_t EnumMax< ErrorCode >()
Definition: NvOnnxParser.h:80
def parse_args(parser)
Definition: train.py:47
def deactivate(self)
Definition: backend/base/runner.py:143
bool mIgnoreBatch
Definition: flattenConcat.h:101
def transformer_layer_opt(prefix, config, init_dict, network, input_tensor, imask)
Definition: builder.py:223
_unsigned
Definition: tensor_quantizer.py:79
int mH
Definition: priorBoxPlugin.h:93
float * getLabels() override
Definition: BatchStream.h:78
Definition: quant_bert.py:1
size_t mLd
Definition: embLayerNormVarSeqlenPlugin.h:93
def test_max_calib(self)
Definition: tensor_quantizer_test.py:178
void setReportableSeverity(Severity reportableSeverity)
Definition: pluginLogging.h:150
bool checkEraseOption(Arguments &arguments, const std::string &option, T &value)
Definition: sampleOptions.cpp:177
IPluginV2 * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: sampleUffPluginV2Ext.cpp:671
const char * INPUT_BLOB_NAME
Definition: sampleCharRNN.cpp:87
int mFileBatchPos
Definition: BatchStream.h:377
@ kSCALE
scale part of IScaleLayer
detail::enable< half &, T >::type operator+=(T rhs)
Arithmetic assignment.
Definition: ieee_half.h:1127
def _run_strip_accents(self, text)
Definition: tokenization.py:250
int mNumber
The number to classify.
Definition: sampleOnnxMNIST.cpp:73
DimsExprs getOutputDimensions(int outputIndex, const nvinfer1::DimsExprs *inputs, int nbInputs, nvinfer1::IExprBuilder &exprBuilder) override
Definition: instanceNormalizationPlugin.cpp:148
void serialize(void *buffer) const override
Serialize the layer.
Definition: proposalPlugin.cpp:205
half & operator*=(float rhs)
Arithmetic assignment.
Definition: samples/common/half.h:1506
outputs
Definition: graph.py:116
int backgroundLabelId
Definition: maskRCNNKernels.h:86
def decode(self, decoder_input, attention_hidden, attention_cell, decoder_hidden, decoder_cell, attention_weights, attention_weights_cum, attention_context, memory, processed_memory, mask)
Definition: demo/Tacotron2/tacotron2/model.py:389
static const int kIMG_H
Definition: sampleFasterRCNN.cpp:88
int64_t mFirstCalibBatch
Definition: sampleConfig.h:49
def __eq__(self, other)
Definition: tensor.py:140
bool isInitialized()
Definition: caffeWeightFactory.cpp:189
max_reldiff
Definition: compare.py:40
float mIOUThreshold
Definition: proposalLayerPlugin.h:100
def forward(self, hidden_states, attention_mask=None, head_mask=None, encoder_hidden_states=None, encoder_attention_mask=None)
Definition: quant_bert.py:83
def max(self, *args)
Definition: 08_replacing_a_subgraph/generate.py:27
def collect(self, x)
Definition: histogram.py:58
Layer type for getting shape of a tensor.
Definition: NvInfer.h:4044
bool processInput(const samplesCommon::BufferManager &buffers, const std::string &inputTensorName, int inputFileIdx) const
Reads the input and mean data, preprocesses, and stores the result in a managed buffer.
Definition: sampleAlgorithmSelector.cpp:444
std::map< std::string, int > mTokenToId
Definition: vocabulary.h:72
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: batchedNMSPlugin.cpp:97
def test_defaults(self, identity_builder_network)
Definition: trt/test_loader.py:172
lstm
Definition: demo/Tacotron2/tacotron2/model.py:194
bool build()
Builds both engines.
Definition: sampleDynamicReshape.cpp:109
void parse(Arguments &arguments) override
Definition: sampleOptions.cpp:444
bool getShapesBuild(Arguments &arguments, std::unordered_map< std::string, ShapeRange > &shapes, const char *argument, nvinfer1::OptProfileSelector selector)
Definition: sampleOptions.cpp:242
TrtCudaStream & getStream(StreamType t)
Definition: sampleInference.cpp:387
static const float RPN_BBOX_STD_DEV[]
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:36
~DataWriter() override=default
@ kLINEAR
Row major linear format.
Definition: quant_conv.py:1
Definition: pyramidROIAlignPlugin.h:36
enable< bool, T, U >::type operator!=(T x, U y)
Comparison for inequality.
Definition: ieee_half.h:2134
nvinfer1::ILayer * parseEltwise(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &, BlobNameToTensor &tensors)
Definition: parseEltwise.cpp:23
float inEnd
Definition: sampleReporting.h:95
bool prepareInput(const samplesCommon::BufferManager &buffers)
Reads the ppm input image, preprocesses, and stores the result in a managed buffer.
Definition: sampleINT8API.cpp:408
def parse_command_line_arguments(args=None)
Definition: mrcnn_to_trt_single.py:36
const char * getPluginVersion() const override
Return the plugin version.
Definition: pyramidROIAlignPlugin.cpp:122
float variance[4]
Definition: NvInferPluginUtils.h:97
@ kREVERSE
Output value is concatenation of values of tensor for each iteration, in reverse order.
def convert_to_ascii(text)
Definition: cleaners.py:79
virtual void addToModel(nvinfer1::INetworkDefinition *network, nvinfer1::ITensor *input, nvinfer1::ITensor **outputLogits)=0
add raw logits to the network
~Reorg() override=default
void add(const std::string &name, nvinfer1::ITensor *tensor)
Definition: blobNameToTensor.h:31
std::vector< float > mCurrentLikelihoods
Definition: beamSearchPolicy.h:70
Functor to enqueue inference from CUDA Graph.
Definition: sampleInference.cpp:233
@ kSUB
Substract the second element from the first.
int getBatchesRead() const override
Definition: BatchStream.h:83
provides a unique 128-bit identifier, which along with the input and output information denotes the v...
Definition: NvInfer.h:6202
float mIOUThreshold
Definition: proposalLayerPlugin.h:137
graph
Definition: graph.py:46
cudnnTensorDescriptor_t mSrcDescriptor
Definition: samples/opensource/samplePlugin/fcPlugin.h:317
PluginFieldType
Definition: NvInferRuntimeCommon.h:826
static TestAtom defineTest(const std::string &name, const std::string &cmdline)
Define a test for logging.
Definition: samples/common/logging.h:297
dtype
Definition: backend/trt_legacy.py:156
Definition: demo/BERT/infer_c/common.h:26
constexpr int32_t EnumMax< DeviceType >()
Maximum number of elements in DeviceType enum.
Definition: NvInferRuntime.h:714
def test_weight_fake_quant_per_tensor(self)
Definition: quant_conv_transposed_test.py:355
constexpr int32_t EnumMax< EngineCapability >()
Maximum number of elements in EngineCapability enum.
Definition: NvInferRuntime.h:60
expr exp(half arg)
Exponential function.
Definition: ieee_half.h:2326
const char * RESIZE_PLUGIN_NAME
Definition: resizeNearestPlugin.cpp:33
int W
Definition: normalizePlugin.h:103
def verbose(self, message, mode=LogMode.EACH)
Definition: Polygraphy/polygraphy/logger/logger.py:262
float half2float_impl(uint16 value, float, true_type)
Convert half-precision to IEEE single-precision.
Definition: ieee_half.h:746
def evaluate_onnx(onnx_filename, data_loader, criterion, print_freq)
Definition: classification_flow.py:276
def build_engine(batch_sizes, workspace_size, sequence_lengths, config, weights_dict, squad_json, vocab_file, calibrationCacheFile, calib_num)
Definition: builder.py:583
constexpr int32_t EnumMax< ScaleMode >()
Maximum number of elements in ScaleMode enum.
Definition: NvInfer.h:2077
def compare(a, b, rtol=1e-7, atol=1e-6, ctol=1e-6)
Definition: tools/pytorch-quantization/tests/utils.py:72
def warning(self, message, mode=LogMode.EACH)
Definition: Polygraphy/polygraphy/logger/logger.py:278
RefineNMSParameters mParam
Definition: detectionLayerPlugin.h:103
new_bytes
Definition: test_cuda.py:30
def forward(ctx, inputs, amax, num_bits=8, unsigned=False, narrow_range=True)
Definition: tensor_quant.py:306
std::shared_ptr< DataReader > ptr
Definition: dataReader.h:34
virtual LikelihoodCombinationOperator::ptr getLikelihoodCombinationOperator() const =0
~ReorgPluginCreator() override=default
def try_match_shape(arr, shape)
Definition: Polygraphy/polygraphy/util/misc.py:325
_model_bytes
Definition: backend/trt/loader.py:120
InstanceNormalizationPluginCreator()
Definition: instanceNormalizationPlugin.cpp:325
stft
Definition: tacotron2/data_function.py:38
static half cast_impl(U arg, true_type)
Definition: samples/common/half.h:2628
expr atan(half arg)
Arc tangent function.
Definition: ieee_half.h:2452
void serialize(void *buffer) override
Serialize the layer.
Definition: debugUtil.cpp:118
virtual int32_t getNbOutputs() const =0
Get the number of outputs of a layer.
PluginTensorDesc desc
Information required to interpret a pointer to tensor data, except that desc.dims has -1 in place of ...
Definition: NvInferRuntime.h:390
virtual IPluginV2 * clone() const =0
Clone the plugin object.
virtual void setReshapeDimensions(Dims dimensions)=0
Set the reshaped dimensions.
Definition: quant_instancenorm_test.py:149
virtual void destroy()=0
Destroy this object.
std::string mNamespace
Definition: sampleUffPluginV2Ext.cpp:689
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: priorBoxPlugin.cpp:168
unsigned char fused_multihead_attention_fp16_384_64_kernel_sm75_cu_o[]
Definition: fused_multihead_attention_fp16_384_64_kernel.sm75.cpp:20
def init_distributed(args, world_size, rank, group_name)
Definition: train.py:158
int64_t mCalibBatchSize
Definition: sampleConfig.h:47
~GridAnchorBasePluginCreator() override=default
int32_t & w()
Get the width.
Definition: NvInfer.h:144
bool mClipBoxes
Definition: batchedNMSPlugin.h:135
Application-implemented logging interface for the builder, engine and runtime.
Definition: NvInferRuntimeCommon.h:1076
def __bool__(self)
Definition: compare.py:43
Context for executing inference using an engine, with functionally unsafe features.
Definition: NvInferRuntime.h:1583
bool isnan(half arg)
Check for NaN.
Definition: ieee_half.h:2744
static void deserialize(void const **buffer, size_t *buffer_size, std::vector< T > *value)
Definition: serialize.hpp:102
xy_t mPooledSize
Definition: multilevelCropAndResizePlugin.h:94
Definition: multilevelCropAndResizePlugin.h:105
static std::vector< PluginField > mPluginAttributes
Definition: resizeNearestPlugin.h:116
def test_print_tensor_quantizer(self)
Definition: print_test.py:39
nvinfer1::DimsHW sortClassScoreDims
Definition: maskRCNNKernels.h:132
CUDADriverWrapper()
Definition: cudaDriverWrapper.cpp:41
Vocabulary::ptr mVocabulary
Definition: textReader.h:46
std::vector< MaskRCNNUtils::BBoxInfo > decodeOutput(const int imageIdx, void *detectionsHost, void *masksHost)
Definition: sampleUffMaskRCNN.cpp:523
void fetchOutputData()
Definition: sampleInference.cpp:376
wrapper for the pinned host memory region
Definition: pinnedHostBuffer.h:32
int64_t packed_mask_stride_in_bytes
Definition: fused_multihead_attention.h:60
void destroy() override
Destroy the plugin object.
Definition: nvFasterRCNNPlugin.cpp:231
def test_save_pb(self)
Definition: tf/test_loader.py:61
IPluginV2 * clone() const override
Clone the plugin object.
Definition: lReluPlugin.cpp:125
const char * getPluginVersion() const override
Return the plugin version.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:280
Definition: fused_multihead_attention.h:292
def o(self, consumer_idx=0, tensor_idx=0)
Definition: tensor.py:112
def narrow_range(self)
Definition: tensor_quantizer.py:147
@ kYOLOREORG
YOLO Reorg Plugin.
def custom_fc(config, network, input_tensor, out_dims, W)
Definition: builder.py:213
DataType type() const noexcept override
The type of the memory that was allocated.
Definition: samples/common/common.h:318
std::map< int32_t, std::vector< int32_t > > userToItemsMap
Definition: sampleMovieLens.cpp:72
std::map< Segment_t, int > Count_t
Definition: bleuScoreWriter.cpp:33
Definition: quant_linear.py:1
def freeze_parameters(model, patterns)
Definition: helper.py:98
virtual void reduceVerbosity()
Decrease verbosity Level.
Definition: parserOnnxConfig.h:103
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleGoogleNet.cpp:157
std::shared_ptr< CudaBind< int > > mValidCnt
Definition: generateDetectionPlugin.h:102
int h
Definition: fused_multihead_attention_v2.h:56
void hostToDevice(TrtCudaStream &stream)
Definition: sampleDevice.h:333
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: nvFasterRCNNPlugin.cpp:143
const char * getPluginName() const override
Return the plugin name.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:275
void allocate(size_t size)
Definition: sampleDevice.h:316
def test_repr(self)
Definition: calibrator_test.py:192
float ** mDeviceScores
Definition: multilevelProposeROIPlugin.h:109
@ kCEIL_DIV
Division rounding up.
std::vector< nvinfer1::PluginField > parsePriorBoxParam(const trtcaffe::LayerParameter &msg, CaffeWeightFactory &weightFactory, BlobNameToTensor &tensors)
Definition: caffeParser.cpp:90
def __init__(self, shape, dtype=trt.int32)
Definition: perf_varseqlen.py:29
def warning(self, message, mode=LogMode.EACH)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:235
static nvinfer1::PluginFieldCollection mFC
Definition: qkvToContextInt8InterleavedPlugin.h:123
attrs
Definition: node.py:39
Definition: qkvToContextInt8InterleavedPlugin.h:33
intermediate_size
Definition: builder.py:94
Definition: multilevelCropAndResizePlugin.h:36
allocated_nbytes
Definition: cuda.py:124
bool isValid(int s) const
Definition: fused_multihead_attention.h:253
input
Definition: 05_folding_constants/generate.py:26
bool builtin_isinf(T arg)
Check for infinity.
Definition: ieee_half.h:404
std::thread makeThread(const InferenceOptions &inference, InferenceEnvironment &iEnv, SyncStruct &sync, int thread, int streamsPerThread, int device, std::vector< InferenceTrace > &trace)
Definition: sampleInference.cpp:548
Definition: fused_multihead_attention_v2.h:25
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: nvFasterRCNNPlugin.cpp:255
T conf_score
Definition: bboxUtils.h:42
nvinfer1::DataType dataType
Definition: sampleUtils.h:220
@ kCALIBRATE_BEFORE_FUSION
Run int8 calibration pass before layer fusion.
const char * getPluginVersion() const override
Return the plugin version.
Definition: batchedNMSPlugin.cpp:370
def _get_amax(self, inputs)
Definition: tensor_quantizer.py:258
std::string weightFileName
Definition: sampleCharRNN.cpp:122
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: proposalPlugin.cpp:265
Definition: gridAnchorPlugin.h:127
Definition: test_cuda.py:24
iterations
Definition: comparator/data_loader.py:87
Definition: denoiser.py:1
def promote_function(fn)
Definition: amp_wrapper.py:28
void serialize(void *buffer) const override
Serialize the layer.
Definition: split.h:62
win_length
Definition: stft.py:48
Definition: quant_pooling.py:117
static std::vector< PluginField > mPluginAttributes
Definition: pyramidROIAlignPlugin.h:124
constexpr int32_t EnumMax< ErrorCode >()
Maximum number of elements in ErrorCode enum.
Definition: NvInferRuntimeCommon.h:1207
unsigned int fused_multihead_attention_v2_fp16_384_64_kernel_sm75_cubin_len
Definition: fused_multihead_attention_v2_fp16_384_64_kernel.sm75.cpp:4700
def _fb_fake_quant(self, inputs, amax)
Definition: tensor_quantizer.py:278
List[int] get_onnx_tensor_shape(Union[onnx.ValueInfoProto, onnx.TensorProto] onnx_tensor)
Definition: onnx_importer.py:46
def __init__(self, num_bits=8, name=None, **kwargs)
Definition: tensor_quant.py:71
std::string mPluginNamespace
Definition: regionPlugin.h:99
def test_export_constant_tensor_to_tensor_proto(self)
Definition: test_exporters.py:34
constexpr int32_t EnumMax< FillOperation >()
Maximum number of elements in FillOperation enum.
Definition: NvInfer.h:4739
int64_t packed_mask_stride_in_bytes
Definition: fused_multihead_attention_v2.h:37
unsigned char fused_multihead_attention_fp16_64_64_kernel_sm75_cu_o[]
Definition: fused_multihead_attention_fp16_64_64_kernel.sm75.cpp:19
void addVerbosity()
Increase the Verbosity Level.
Definition: sampleConfig.h:117
~PyramidROIAlign() override=default
ICudaEngine * loadEngine(const std::string &engine, int DLACore, std::ostream &err)
Load a serialized engine.
Definition: sampleEngines.cpp:632
static const int POST_NMS_ROIS_INFERENCE
Definition: tlt_mrcnn_config.h:83
void transferInputToDevice(TrtCudaStream &stream)
Definition: sampleUtils.h:339
SFINAE helper for generic half-precision functions.
Definition: ieee_half.h:372
def __init__(self, old, old_size, new, new_size)
Definition: test_cuda.py:26
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: proposalLayerPlugin.cpp:362
def __init__(self, shape=None, dtype=None)
Definition: cuda.py:114
int initialize() override
Initialize the layer for execution.
Definition: instanceNormalizationPlugin.cpp:155
bool infer()
Definition: sampleUffMaskRCNN.cpp:422
int32_t expectedPredictedMaxRatingItem
Definition: sampleMovieLensMPS.cpp:158
def test_i_multiple_inputs(self)
Definition: test_ir.py:86
IOnnxConfig * createONNXConfig()
def initialize(float_module_list=None, custom_quant_modules=None)
Definition: quant_modules.py:110
def synchronize(self)
Definition: cuda.py:99
def __init__(self, in_channels, out_channels, kernel_size, stride, padding, dilation, transposed, output_padding, groups, bias, padding_mode, quant_desc_input, quant_desc_weight)
Definition: quant_conv.py:67
LogStreamConsumer gLogFatal
Definition: logger.cpp:27
std::vector< nvinfer1::IPluginV2 * > mNewPlugins
Definition: caffeParser.h:90
IPluginV2Ext * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: proposalPlugin.cpp:434
~ResizeNearest() override=default
int getBatchSize() const override
Definition: EntropyCalibrator.h:111
def register_callback(self, callback)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:123
Definition: quant_pooling_test.py:128
nvinfer1::ILayer * parsePReLU(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &weightFactory, BlobNameToTensor &tensors)
Definition: parsePReLU.cpp:23
def test_quant_module_replacement(self)
Definition: integration_test.py:101
The SampleMLPParams structure groups the additional parameters required by the MLP sample.
Definition: sampleMLP.cpp:45
uint32_t QuantizationFlags
Represents a collection of one or more QuantizationFlag values using binary OR operations.
Definition: NvInfer.h:6345
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: multilevelProposeROIPlugin.cpp:236
strict_types
Definition: backend/trt/loader.py:327
IPluginV2Ext * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: reorgPlugin.cpp:217
Definition: InferPlugin.cpp:69
ElemType * raw() noexcept
Definition: samples/common/common.h:348
int initialize() override
Initialize the layer for execution.
Definition: embLayerNormVarSeqlenPlugin.cpp:352
ActivationType
Enumerates the types of activation to perform in an activation layer.
Definition: NvInferRuntimeCommon.h:108
@ kFP16
Enable FP16 layer selection, with FP32 fallback.
virtual Dims getOutputDimensions(int32_t index, const Dims *inputs, int32_t nbInputDims)=0
Get the dimension of an output tensor.
int64_t mLabel
Definition: sampleConfig.h:44
virtual bool isNetworkInput() const =0
Whether the tensor is a network input.
const char * getPluginVersion() const override
Return the plugin version.
Definition: embLayerNormVarSeqlenPlugin.cpp:342
def forward(self, input)
Definition: quant_pooling.py:148
def forward(self, input, output_size=None)
Definition: quant_conv.py:337
def prenet_infer(self, x)
Definition: export_tacotron2_onnx.py:92
WeightType
Definition: weightType.h:22
~SLPEmbedder() override=default
virtual int32_t getMissing(int32_t size, const char **layerNames, WeightsRole *roles)=0
Get description of missing weights.
def __init__(self, attention_rnn_dim, embedding_dim, attention_dim, attention_location_n_filters, attention_location_kernel_size)
Definition: demo/Tacotron2/tacotron2/model.py:49
def __init__(self, explicit_precision, explicit_batch=None)
Definition: backend/trt/loader.py:95
void convert(nvinfer1::Weights &weights, nvinfer1::DataType targetType)
Definition: caffeWeightFactory.cpp:160
def test_export_graph(self, model)
Definition: test_exporters.py:127
runtime
Definition: comparator/struct.py:45
void * data()
Returns pointer to underlying array.
Definition: buffers.h:112
void setPluginNamespace(const char *libNamespace) override
Set the namespace that this plugin object belongs to.
Definition: batchTilePlugin.cpp:173
conv2
Definition: tools/pytorch-quantization/tests/fixtures/models.py:48
def forward(self, model_output, targets)
Definition: tacotron2/loss_function.py:24
constexpr auto param_type
Definition: skipLayerNormInt8InterleavedPlugin.cpp:51
static int reportPass(const TestAtom &testAtom)
Definition: samples/common/logging.h:347
int inputHeight
Definition: sampleUffFasterRCNN.cpp:69
Definition: quant_pooling_test.py:38
The TensorRT ONNX parser API namespace.
Definition: NvOnnxConfig.h:22
def main()
Definition: inference_trt.py:288
device_inputs
Definition: demo/BERT/helpers/calibrator.py:46
int32_t length
Number of data entries in the Plugin attribute.
Definition: NvInferRuntimeCommon.h:865
@ kLOOP_OUTPUT
Loop output layer.
def parse_args(parser)
Definition: copyright-scan.py:99
std::unordered_map< std::string, nvinfer1::IPluginCreator * > mPluginRegistry
Definition: caffeParser.h:91
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleINT8API.cpp:80
~NormalizePluginCreator() override=default
int getAttentionSize() override
get the size of the attention vector
Definition: slpAttention.cpp:68
SampleCharRNNWeightNames weightNames
Definition: sampleCharRNN.cpp:128
int mPreNMSTopK
Definition: multilevelProposeROIPlugin.h:95
virtual IMatrixMultiplyLayer * addMatrixMultiply(ITensor &input0, MatrixOperation op0, ITensor &input1, MatrixOperation op1)=0
Add a MatrixMultiply layer to the network.
std::vector< int > inputDims
Definition: sampleINT8API.cpp:48
void resize(size_t newSize)
Resizes the buffer.
Definition: buffers.h:144
constexpr int32_t EnumMax< TensorLocation >()
Maximum number of elements in TensorLocation enum.
Definition: NvInferRuntimeCommon.h:1023
half abs(half arg)
Absolute value.
Definition: ieee_half.h:2228
def test_two_tensor(self, verbose)
Definition: calibrator_test.py:171
def __call__(self)
Definition: backend/trt_legacy.py:54
Definition: quant_rnn_test.py:512
path
Definition: onnx_models.py:36
bool readASCIIFile(const std::string &fileName, const size_t size, std::vector< T > &out)
Definition: samples/common/common.h:451
const char * mModelStreamFd
Definition: sampleMovieLensMPS.cpp:151
def forward(self, x)
Definition: tools/pytorch-quantization/tests/fixtures/models.py:52
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: generateDetectionPlugin.cpp:303
def to_constant(self, np.ndarray values)
Definition: tensor.py:59
def __init__(self, model, name=None)
Definition: backend/onnx/runner.py:29
virtual const char * getPluginName() const =0
Return the plugin name.
TimePoint cpuStart
Definition: sampleInference.cpp:171
Definition: backend/onnxrt/loader.py:22
def __init__(self, path)
Definition: backend/common/loader.py:20
UffPoolPluginV2(const PluginFieldCollection &fc)
Definition: sampleUffPluginV2Ext.cpp:340
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:1
Legacy calibrator left for backward compatibility with TensorRT 2.0.
Definition: NvInfer.h:6108
def toposort(self)
Definition: graph.py:254
nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType *inputType, int nbInputs) const override
Definition: batchedNMSPlugin.cpp:439
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: split.h:120
def _expand_dollars(m)
Definition: numbers.py:39
def test_repr(self)
Definition: calibrator_test.py:245
static bool isnotequal(half x, half y)
Comparison implementation.
Definition: samples/common/half.h:2398
IPluginV2Ext * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: batchedNMSPlugin.cpp:627
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: priorBoxPlugin.cpp:343
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: samplePlugin.cpp:107
GenericBuffer< HostAllocator, HostFree > HostBuffer
Definition: buffers.h:217
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: normalizePlugin.cpp:196
fc2
Definition: tools/pytorch-quantization/tests/fixtures/models.py:32
Definition: tools/pytorch-quantization/tests/fixtures/__init__.py:1
sample::Logger::TestResult infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleINT8API.cpp:590
void serializeFromDevice(char *&hostBuffer, Weights deviceWeights) const
Definition: priorBoxPlugin.cpp:222
_calib_bin_edges
Definition: histogram.py:49
def __str__(self)
Definition: compare.py:53
float e2e
Definition: sampleReporting.h:56
A convolution layer in a network definition.
Definition: NvInfer.h:1199
void throwCudaError(const char *file, const char *function, int line, int status, const char *msg)
Definition: checkMacrosPlugin.cpp:54
def test_fake_quant_per_channel(self)
Definition: quant_conv_test.py:145
Plugin class for user-implemented layers.
Definition: NvInferRuntime.h:223
static const int DETECTION_MAX_INSTANCES
Definition: tlt_mrcnn_config.h:40
std::string getInfo() override
get the textual description of the component
Definition: slpProjection.cpp:64
virtual ~DeviceBuffer()
Definition: deviceBuffer.h:38
constexpr int defaultDevice
Definition: sampleOptions.h:41
DetectionLayer(int num_classes, int keep_topk, float score_threshold, float iou_threshold)
Definition: detectionLayerPlugin.cpp:97
CUresult(* _cuLaunchKernel)(CUfunction f, unsigned int gridDimX, unsigned int gridDimY, unsigned int gridDimZ, unsigned int blockDimX, unsigned int blockDimY, unsigned int blockDimZ, unsigned int sharedMemBytes, CUstream hStream, void **kernelParams, void **extra)
Definition: cudaDriverWrapper.h:83
def parse_output(self, outputs, output_lengths)
Definition: demo/Tacotron2/tacotron2/model.py:635
def activate_impl(self)
Definition: backend/onnxrt/runner.py:39
virtual void clearErrors()=0
Clear errors from prior calls to parse.
float mEpsilon
Definition: groupNormalizationPlugin.h:98
audiopaths_and_text
Definition: tacotron2/data_function.py:33
nvinfer1::IPluginV2DynamicExt * clone() const override
Clone the plugin object.
Definition: split.h:98
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: resizeNearestPlugin.cpp:57
nvinfer1::IPluginV2DynamicExt * clone() const override
Clone the plugin object.
Definition: groupNormalizationPlugin.cpp:193
int d
Definition: fused_multihead_attention.h:79
def test_single_key(self, resnet18)
Definition: optim_helper_test.py:53
nvinfer1::ILayer * parseSoftMax(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &, BlobNameToTensor &tensors)
Definition: parseSoftMax.cpp:23
bool mChannelShared
Definition: normalizePlugin.h:127
Structure containing plugin attribute field names and associated data This information can be parsed ...
Definition: NvInferRuntimeCommon.h:846
int mS
Definition: sampleUffPluginV2Ext.cpp:179
@ kSAME_UPPER
Use SAME padding, with prePadding <= postPadding.
const Dims int32_t
Definition: NvInferRuntime.h:554
Definition: likelihoodCombinationOperator.h:24
def __init__(self, num_bits, axis, unsigned)
Definition: tools/pytorch-quantization/pytorch_quantization/calib/calibrator.py:31
int numPriors
Definition: batchedNMSPlugin.h:70
cudaStream_t mStream
Definition: sampleMovieLensMPS.cpp:247
def test_reduce_amax(self)
Definition: quant_utils_test.py:32
root_dir
Definition: Polygraphy/polygraphy/logger/logger.py:128
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: groupNormalizationPlugin.cpp:270
int initialize() override
Initialize the layer for execution.
Definition: nvFasterRCNNPlugin.cpp:117
virtual ~IConcatenationLayer()
Definition: NvInfer.h:2251
IPluginV2Ext * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: priorBoxPlugin.cpp:482
void destroy() override
Destroy the plugin object.
Definition: coordConvACPlugin.cpp:139
void destroy() override
Destroy the plugin object.
Definition: flattenConcat.cpp:279
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: flattenConcat.cpp:316
def set_default_quant_desc_input(cls, value)
Definition: _utils.py:37
const std::string mLayerName
Definition: batchTilePlugin.h:79
def question_features(tokens, question)
Definition: inference_c.py:117
constexpr size_t xmmasM384
Definition: embLayerNormVarSeqlenPlugin.cpp:39
def map_node_outputs(graphdef)
Definition: tf/util.py:56
void serialize(void *buffer) const override
Serialize the layer.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:239
bool teardown()
Used to clean up any state created in the sample class.
Definition: sampleAlgorithmSelector.cpp:615
def test_device_buffer_free(self)
Definition: test_cuda.py:79
Definition: backend/tf/loader.py:307
GridAnchorRectPluginCreator()
Definition: gridAnchorPlugin.cpp:502
int inputChannels
Definition: sampleUffFasterRCNN.cpp:68
Definition: quant_pooling.py:89
def __init__(self, network_loader=None, max_workspace_size=None, max_batch_size=None, fp16=None, tf32=None, load_engine=None, save_engine=None, layerwise=False, plugins=[], name=None)
Definition: backend/trt_legacy.py:187
void printPerformanceReport(const std::vector< InferenceTrace > &trace, const ReportingOptions &reporting, float warmupMs, int queries, std::ostream &os)
Print and summarize a timing trace.
Definition: sampleReporting.cpp:195
Opaque handle that holds logging information for a particular test.
Definition: pluginLogging.h:287
def identity_model()
Definition: onnx_models.py:92
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: regionPlugin.cpp:494
float * _d_scale
Definition: instanceNormalizationPlugin.h:94
calculates softmax likelihood and TopK indices for the raw input logits
Definition: softmaxLikelihood.h:30
SpecialSlicePluginCreator()
Definition: specialSlicePlugin.cpp:35
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: proposalPlugin.cpp:349
Definition: modules/quant_rnn.py:39
void find_divisor(int denom, unsigned int &mul_coeff, unsigned int &shift_coeff)
Definition: reducedMathPlugin.cpp:49
const char * getPluginNamespace() const override
Return the namespace of the plugin creator object.
Definition: batchTilePlugin.h:104
void inferenceExecution(const InferenceOptions &inference, InferenceEnvironment &iEnv, SyncStruct &sync, int offset, int streams, int device, std::vector< InferenceTrace > &trace)
Definition: sampleInference.cpp:507
def __eq__(self, other)
Definition: node.py:116
def __init__(self, in_dim, sizes)
Definition: demo/Tacotron2/tacotron2/model.py:109
unsigned int fused_multihead_attention_v2_int8_128_64_kernel_sm80_cubin_len
Definition: fused_multihead_attention_v2_int8_128_64_kernel.sm80.cpp:14791
p_decoder_dropout
Definition: export_tacotron2_onnx.py:111
def forward(self, input, hx=None)
Definition: modules/quant_rnn.py:361
def test_multi_keys(self, resnet18)
Definition: optim_helper_test.py:39
void serialize(void *buffer) const override
Serialize the layer.
Definition: cropAndResizePlugin.cpp:147
is_active
Definition: backend/base/runner.py:47
SampleUniquePtr< nvinfer1::ICudaEngine > mEngine
Definition: sampleUffPluginV2Ext.cpp:333
Definition: backend/tf/loader.py:212
TFusedMultiHeadAttentionXMMAKernel(const TKernelMeta *pMetaStart, unsigned int nMetaCount, Data_type type, unsigned int sm)
Definition: fused_multihead_attention.h:203
void launch_large(cudaStream_t stream, const int ld, const int total, const int8_t *input, const int8_t *skip, const half *beta, const half *gamma, int8_t *output, const float dqScaleIn, const float dqScaleSkip, const float qScale)
std::string dynamicRangeFileName
Definition: sampleINT8API.cpp:654
cond_layers
Definition: demo/Tacotron2/waveglow/model.py:97
@ kVERBOSE
Verbose messages with debugging information.
unsigned char fused_multihead_attention_v2_int8_192_64_kernel_sm80_cubin[]
Definition: fused_multihead_attention_v2_int8_192_64_kernel.sm80.cpp:19
tacotron2
Definition: export_tacotron2_onnx.py:66
std::vector< std::shared_ptr< CudaBind< float > > > mTempBboxes
Definition: multilevelProposeROIPlugin.h:108
constexpr int defaultDuration
Definition: sampleOptions.h:48
TRTParsedArgs parseArgs(int argc, const char *const *argv, const std::vector< TRTOption > &options)
parseArgs parses an argument list and returns a TRTParsedArgs with the fields set accordingly.
Definition: getOptions.cpp:142
def test_fake_quant_per_tensor(self)
Definition: quant_conv_test.py:344
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: nmsPlugin.cpp:251
const char * getPluginVersion() const override
Return the plugin version.
Definition: specialSlicePlugin.cpp:47
static const std::string mSosStr
Definition: vocabulary.h:68
void putOutput()
Definition: pluginLogging.h:68
Definition: serialize.hpp:39
DetectionLayerPluginCreator()
Definition: detectionLayerPlugin.cpp:35
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: gridAnchorPlugin.cpp:302
virtual void setTextFileName(const char *textFilename)
Set the File Name of the Network Description as a Text File.
Definition: parserOnnxConfig.h:116
a functional part of the sample
Definition: component.h:30
TrtCudaStream()
Definition: sampleDevice.h:60
Definition: tools/pytorch-quantization/tests/fixtures/models.py:1
TensorFormat format
Definition: NvInferRuntimeCommon.h:371
def onnx_to_trt_name(onnx_name)
Definition: builder_varseqlen.py:418
nvinfer1::DimsHW sortClassPosDims
Definition: maskRCNNKernels.h:171
def allocate(self, nbytes)
Definition: cuda.py:133
Definition: sampleDevice.h:281
def __init__(self)
Definition: quant_modules.py:53
def test_serialized_engine_loader_from_buffer(self, identity_engine)
Definition: trt/test_loader.py:83
void destroy() override
Destroy the plugin object.
Definition: specialSlicePlugin.h:53
static std::basic_istream< charT, traits > & read(std::basic_istream< charT, traits > &in, half &arg)
Input implementation.
Definition: samples/common/half.h:1636
void synchronize()
Definition: sampleDevice.h:83
def constfold(self, graphdef, output_names)
Definition: backend/tf/loader.py:39
def test_fake_quant_per_channel(self)
Definition: quant_conv_transposed_test.py:428
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: nmsPlugin.cpp:188
The SampleMNIST class implements the MNIST sample.
Definition: sampleMNIST.cpp:48
std::vector< Binding > mBindings
Definition: sampleUtils.h:452
std::string errMsg
Definition: getOptions.h:59
std::string mNamespace
Definition: sampleUffPluginV2Ext.cpp:642
int64_t o_stride_in_bytes
Definition: fused_multihead_attention.h:62
int label
Definition: sampleUffMaskRCNN.cpp:65
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: multilevelProposeROIPlugin.cpp:64
PoolingType pType
Definition: sampleUffPluginV2Ext.cpp:185
bool mReadCache
Definition: EntropyCalibrator.h:91
std::string networkName
The name of the network.
Definition: sampleINT8.cpp:51
Definition: backend/trt/loader.py:258
def __call__(self, tensor)
Definition: graph.py:34
Definition: onnx/test_loader.py:29
Implements Entropy calibrator 2. CalibrationAlgoType is kENTROPY_CALIBRATION_2.
Definition: EntropyCalibrator.h:102
void processShapes(std::unordered_map< std::string, ShapeRange > &shapes, bool minShapes, bool optShapes, bool maxShapes, bool calib)
Definition: sampleOptions.cpp:274
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleINT8API.cpp:106
Definition: samples/opensource/sampleUffFasterRCNN/config.py:1
def test_disable(self)
Definition: tensor_quantizer_test.py:132
def test_scaled_mode(self)
Definition: tensor_quant_test.py:271
std::vector< samplesCommon::PPM< 3, 300, 300 > > mPPMs
PPMs of test images.
Definition: sampleUffSSD.cpp:95
LogStreamConsumerBuffer(LogStreamConsumerBuffer &&other)
Definition: pluginLogging.h:42
@ kPERMUTE
Permute plugin.
std::vector< std::string > plugins
Definition: sampleOptions.h:139
~InstanceNormalizationPluginCreator() override=default
Definition: tensor_quant.py:1
void allocateChunk(T *&ptr, int count)
Definition: regionPlugin.cpp:41
ElementWiseOperation
Enumerates the binary operations that may be performed by an ElementWise layer.
Definition: NvInfer.h:2605
constexpr const char * SPLIT_PLUGIN_NAME
Definition: split.h:31
Layer type for shuffling data.
Definition: NvInfer.h:3749
opset
Definition: graph.py:122
bool verifyOutput(const samplesCommon::BufferManager &buffers, const std::string &outputTensorName, int groundTruthDigit) const
Verifies that the output is correct and prints it.
Definition: sampleMNIST.cpp:206
std::string saveEngine
Definition: sampleCharRNN.cpp:124
def node_ids(self)
Definition: graph.py:158
bool teardown()
Used to clean up any state created in the sample class.
Definition: sampleCharRNN.cpp:948
def __setstate__(self, d)
Definition: modules/quant_rnn.py:244
bool refittable
Definition: sampleOptions.h:113
def input_quantizer(self)
Definition: _utils.py:88
std::unordered_map< uint64_t, FusedMultiHeadAttentionKernelInfo > mFunctions
Definition: fused_multihead_attention.h:287
@ kRANDOM_UNIFORM
Generate a tensor with random values drawn from a uniform distribution.
@ DATA_TYPE_INT8
Definition: fused_multihead_attention_common.h:30
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: generateDetectionPlugin.cpp:153
int mHeadSize
Definition: qkvToContextInt8InterleavedPlugin.h:84
def test_fold_constants_no_foldable_constants(self)
Definition: test_ir.py:759
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: qkvToContextInt8InterleavedPlugin.cpp:216
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: nmsPlugin.cpp:182
~TrtCudaEvent()
Definition: sampleDevice.h:124
bool prepare()
Prepares the model for inference by creating execution contexts and allocating buffers.
Definition: sampleDynamicReshape.cpp:291
virtual bool enqueueV2(void **bindings, cudaStream_t stream, cudaEvent_t *inputConsumed) noexcept=0
Asynchronously execute inference.
selects the embedding vector from the weight matrix using index provided in the input
Definition: slpEmbedder.h:36
def compress(obj)
Definition: Polygraphy/polygraphy/util/misc.py:187
int nbCalBatches
The number of batches for calibration.
Definition: sampleINT8.cpp:49
def get_mha_dtype(config)
Definition: builder.py:115
const std::map< char, int > charToID
Definition: sampleCharRNN.cpp:99
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: nmsPlugin.cpp:215
bool teardown()
Definition: sampleUffMaskRCNN.cpp:472
bool operator()(void **ptr, size_t size) const
Definition: buffers.h:182
Weights copyToDevice(const void *hostData, size_t count)
void finalize() override
it is called right after inference ends
Definition: benchmarkWriter.cpp:44
bool verifyOutput(const samplesCommon::BufferManager &buffers)
Filters output detections and verify results.
Definition: sampleUffFasterRCNN.cpp:413
Definition: trt/test_loader.py:160
@ kPLUGIN_V2
PluginV2 layer.
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: specialSlicePlugin.cpp:87
@ kDIV
Divide the first element by the second.
def free(self)
Definition: perf.py:35
def squad_output(prefix, config, init_dict, network, input_tensor)
Definition: builder_varseqlen.py:310
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleUffMaskRCNN.cpp:313
size_t getWorkspaceSize(const nvinfer1::PluginTensorDesc *inputs, int nbInputs, const nvinfer1::PluginTensorDesc *outputs, int nbOutputs) const override
Definition: embLayerNormVarSeqlenPlugin.cpp:254
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleINT8.cpp:61
Definition: regionPlugin.h:102
int mNumLayers
Definition: gridAnchorPlugin.h:95
std::vector< float > mData
Definition: BatchStream.h:150
std::vector< nvinfer1::PluginField > parseNormalizeParam(const trtcaffe::LayerParameter &msg, CaffeWeightFactory &weightFactory, BlobNameToTensor &tensors)
Definition: caffeParser.cpp:48
virtual bool supportsFormat(DataType type, PluginFormat format) const =0
Check format support.
def test_append(self, field_names)
Definition: test_ir.py:230
def test_set_outputs_updates_old_outputs(self)
Definition: test_ir.py:171
@ kRELU
Single gate RNN w/ ReLU activation function.
int dataTypeSize(nvinfer1::DataType dataType)
Definition: sampleUtils.h:45
def test_input_fake_quant_disable(self)
Definition: quant_pooling_test.py:276
Definition: tensor.py:149
def dynamic_range_decompression(x, C=1)
Definition: audio_processing.py:103
Definition: backend/pyt/loader.py:18
def test_context(self)
Definition: trt/test_runner.py:51
bool teardown()
Used to clean up any state created in the sample class.
Definition: samplePlugin.cpp:356
const bool mEnableGraph
Definition: bert_infer.h:338
LReluPluginCreator()
Definition: lReluPlugin.cpp:132
const char * getPluginVersion() const override
Return the plugin version.
Definition: lReluPlugin.cpp:145
def __str__(self)
Definition: tensor.py:132
Definition: tf/test_runner.py:26
nvinfer1::Dims mInputDims
The dimensions of the input to the network.
Definition: sampleAlgorithmSelector.cpp:361
Definition: tokenization.py:1
def test_can_load_libnvinfer_plugins(self)
Definition: trt/test_loader.py:63
@ kNC2HW2
Deprecated name of kCHW2, provided for backwards compatibility and will be removed in TensorRT 8....
virtual void setMaxWorkspaceSize(std::size_t workspaceSize)=0
Set the maximum workspace size.
std::string mPluginName
Definition: gridAnchorPlugin.h:120
std::unordered_multimap< std::string, std::string > Arguments
Definition: sampleOptions.h:63
constexpr int32_t EnumMax< RNNOperation >()
Maximum number of elements in RNNOperation enum.
Definition: NvInfer.h:2801
T type
Definition: ieee_half.h:379
void setReportableSeverity(Severity severity)
Method for controlling the verbosity of logging output.
Definition: samples/common/logging.h:254
Definition: tlt_mrcnn_config.h:25
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: specialSlicePlugin.cpp:132
std::string mPluginNamespace
Definition: gridAnchorPlugin.h:99
def forward(self, input)
Definition: quant_pooling.py:71
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: detectionLayerPlugin.cpp:170
nvinfer1::Dims getDims() const
Definition: frcnnUtils.h:319
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: split.h:124
std::shared_ptr< Context > ptr
Definition: contextNMT.h:35
const char * getPluginName() const override
Return the plugin name.
Definition: specialSlicePlugin.cpp:42
bert::cuda_unique_ptr< float > mGammaDev
Definition: embLayerNormVarSeqlenPlugin.h:88
_calibrator
Definition: tensor_quantizer.py:102
def __call__(self)
Definition: backend/onnx/loader.py:166
def o(self, consumer_idx=0, tensor_idx=0)
Definition: node.py:65
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: multilevelCropAndResizePlugin.cpp:150
Definition: base_exporter.py:1
~Embedder() override=default
@ kINT32
INT32 field type.
std::string mNamespace
Definition: cropAndResizePlugin.h:102
cudaGraphExec_t mExecGraph
Definition: bert_infer.h:355
virtual int64_t getTactic() const =0
Return tactic of the algorithm.
T type
Definition: ieee_half.h:374
Definition: quant_conv.py:220
void setPluginNamespace(const char *libNamespace) override
Set the namespace that this plugin object belongs to.
Definition: proposalLayerPlugin.cpp:182
ScaleMode
Controls how shift, scale and power are applied in a Scale layer.
Definition: NvInfer.h:2068
virtual bool isPluginV2(const char *layerName)=0
A user implemented function that determines if a layer configuration is provided by an IPluginV2.
def __repr__(self)
Definition: graph.py:471
def __eq__(self, rhs)
Definition: tensor_quant.py:170
def forward(self, input)
Definition: quant_pooling.py:136
static const int NUM_CLASSES
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:60
const FusedMultiHeadAttentionXMMAKernelV2 * getXMMAKernelsV2(Data_type type, unsigned int sm)
Definition: fused_multihead_attention_v2.h:522
name
Definition: 09_shape_operations_with_the_layer_api/generate.py:54
dir
Definition: backend/tf/loader.py:175
enable< bool, T, U >::type operator==(T x, U y)
Comparison for equality.
Definition: ieee_half.h:2127
@ kEXPLICIT_ROUND_DOWN
Use explicit padding, rounding output size down.
std::string meanFileName
Filename of mean file of a network.
Definition: argsParser.h:55
void stop()
Definition: samples/common/common.h:825
def attention_layer_opt(prefix, config, init_dict, network, input_tensor, mask_idx, cu_seqlens, max_seqlen)
Definition: builder_varseqlen.py:120
void dumpBindings(const nvinfer1::IExecutionContext &context, bool(*predicate)(const Binding &b), std::ostream &os) const
Definition: sampleUtils.h:401
const std::string mLayerName
Definition: skipLayerNormInt8InterleavedPlugin.h:71
attention_rnn
Definition: demo/Tacotron2/tacotron2/model.py:258
int enqueue(const nvinfer1::PluginTensorDesc *inputDesc, const nvinfer1::PluginTensorDesc *outputDesc, const void *const *inputs, void *const *outputs, void *workspace, cudaStream_t stream) override
Execute the layer.
Definition: groupNormalizationPlugin.cpp:107
def get_input_metadata(graph)
Definition: tf/util.py:83
Definition: quant_pooling.py:76
@ DATA_TYPE_E8M7
Definition: fused_multihead_attention_common.h:26
unsigned int fused_multihead_attention_fp16_96_64_kernel_sm80_cu_o_len
Definition: fused_multihead_attention_fp16_96_64_kernel.sm80.cpp:2345
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: flattenConcat.cpp:221
def test_set_inputs_updates_old_inputs(self)
Definition: test_ir.py:165
float * getLabels()
Definition: frcnnUtils.h:304
def test_multiple_profiles(self, identity_builder_network)
Definition: trt/test_loader.py:234
unsigned short uint16
Unsigned integer of (at least) 16 bits width.
Definition: ieee_half.h:331
def weight_quantizer(self)
Definition: _utils.py:92
def test_unused_kwargs(self)
Definition: quant_linear_test.py:228
bool constructNetwork(SampleUniquePtr< nvinfer1::IBuilder > &builder, SampleUniquePtr< nvinfer1::INetworkDefinition > &network, SampleUniquePtr< nvinfer1::IBuilderConfig > &config, SampleUniquePtr< nvcaffeparser1::ICaffeParser > &parser, DataType dataType)
Parses a Caffe model and creates a TensorRT network.
Definition: sampleINT8.cpp:203
void enableDLA(IBuilder *builder, IBuilderConfig *config, int useDLACore, bool allowGPUFallback=true)
Definition: samples/common/common.h:575
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: pyramidROIAlignPlugin.cpp:247
virtual int32_t getTensorRTVersion() const
Return the version of the API the plugin creator was compiled with.
Definition: NvInferRuntimeCommon.h:896
Definition: samples/opensource/samplePlugin/fcPlugin.h:321
std::vector< void * > mBindings
Definition: bert_infer.h:342
max_workspace_size
Definition: backend/trt/loader.py:321
def __init__(self, vocab_file, do_lower_case=True)
Definition: tokenization.py:163
Register the plugin creator to the registry The static registry object will be instantiated when the ...
Definition: NvInferRuntimeCommon.h:1397
@ kEQUAL
1 if operands are equal, 0 otherwise.
unsigned char fused_multihead_attention_fp16_384_64_kernel_sm80_cu_o[]
Definition: fused_multihead_attention_fp16_384_64_kernel.sm80.cpp:20
Definition: base_importer.py:19
std::string getInfo() override
get the textual description of the component
Definition: textWriter.cpp:40
int iW
Definition: coordConvACPlugin.h:86
def get_dtype(tensor)
Definition: onnx/util.py:132
~Normalize() override=default
virtual const char * getPluginVersion() const =0
Return the plugin version.
def test_print_descriptor(self)
Definition: print_test.py:35
static std::vector< nvinfer1::PluginField > mPluginAttributes
Definition: embLayerNormVarSeqlenPlugin.h:136
Definition: NvUffParser.h:87
@ kOR
Logical OR of two elements.
def test_shape_matching(case)
Definition: test_misc.py:72
Functor to enqueue inference with explict batch.
Definition: sampleInference.cpp:214
std::string engine
Definition: sampleOptions.h:121
Definition: backend/tf/loader.py:27
def test_init_calib(self)
Definition: tensor_quantizer_test.py:168
constexpr int32_t EnumMax< DimensionType >()
Maximum number of elements in DimensionType enum.
Definition: NvInferRuntimeCommon.h:175
static half_float::half denorm_min()
Smallest positive subnormal value.
Definition: samples/common/half.h:4264
def __init__(self, model, path=None)
Definition: backend/onnx/loader.py:154
def copy_state_and_quantize(dst, src, num_bits)
Definition: tools/pytorch-quantization/tests/utils.py:40
static half cast(half arg)
Definition: samples/common/half.h:2684
~DetectionLayerPluginCreator()
Definition: detectionLayerPlugin.h:113
bool reorderSubBuffers(void *input, const int32_t *order, int32_t num, int32_t size)
Takes an input stream and re-orders num chunks of the data given the size and order.
def forward(self, model_output, clean_audio)
Definition: waveglow/loss_function.py:24
expr hypot(half x, half y)
Hypotenuse function.
Definition: ieee_half.h:2394
void * mDeviceKernel
Definition: samples/opensource/samplePlugin/fcPlugin.h:312
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: batchTilePlugin.cpp:91
def stats(self, runner_pair=None)
Definition: comparator/struct.py:194
onnx.NodeProto export_node(Node node)
Definition: onnx_exporter.py:54
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: priorBoxPlugin.cpp:283
float stepW
Definition: NvInferPluginUtils.h:99
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleAlgorithmSelector.cpp:315
int streams
Definition: sampleOptions.h:153
expr asin(half arg)
Arc sine.
Definition: ieee_half.h:2438
int32_t W
Definition: NvInferPluginUtils.h:147
def __init__(self, n_mel_channels, n_flows, n_group, n_early_every, n_early_size, WN_config)
Definition: demo/Tacotron2/waveglow/model.py:158
void addPluginCreator(void *logger, const char *libNamespace)
Definition: InferPlugin.cpp:79
Definition: sampleReformatFreeIO.cpp:64
The SampleParams structure groups the basic parameters required by all sample networks.
Definition: argsParser.h:36
def __init__(self, filter_length=1024, hop_length=256, win_length=1024, n_mel_channels=80, sampling_rate=22050, mel_fmin=0.0, mel_fmax=8000.0)
Definition: layers.py:58
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: groupNormalizationPlugin.cpp:101
def convert_ids_to_tokens(self, ids)
Definition: tokenization.py:180
int inputW
The input width.
Definition: sampleMNISTAPI.cpp:49
def destroy_stream(self, stream)
Definition: cuda.py:55
static std::vector< PluginField > mPluginAttributes
Definition: flattenConcat.h:129
virtual float getDynamicRangeMin(const char *tensorName) const =0
Get minimum of dynamic range.
void destroy() override
Destroy the plugin object.
Definition: split.h:102
virtual IErrorRecorder * getErrorRecorder() const noexcept=0
set the ErrorRecorder assigned to this interface.
Definition: tokenization.py:215
void setMessage(const char *msg)
Definition: checkMacrosPlugin.h:87
nvinfer1::Dims mInputDims
Definition: sampleUffMNIST.cpp:103
hidden_size
Definition: modules/quant_rnn.py:47
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: flattenConcat.cpp:161
ReduceOperation
Enumerates the reduce operations that may be performed by a Reduce layer.
Definition: NvInfer.h:3556
static T cast_impl(half arg, false_type)
Definition: samples/common/half.h:2676
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: proposalPlugin.cpp:235
@ kRELU
Rectified linear activation.
MetadataTuple
Definition: common/struct.py:20
def insert(self, i, x)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:107
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:219
Definition: sampleUtils.h:457
Definition: backend/tf/loader.py:357
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: nmsPlugin.cpp:124
std::unordered_map< std::string, int > getInputBindings() const
Definition: sampleUtils.h:418
const std::string RNNB_L1_NAME
Definition: sampleCharRNN.cpp:76
NMSParameters param
Definition: batchedNMSPlugin.h:67
Definition: instanceNormalizationPlugin.h:32
Definition: backend/trt_legacy.py:143
int getVolume(nvinfer1::Dims dims)
Definition: trtUtil.cpp:36
LogStreamConsumer LOG_VERBOSE(const Logger &logger)
produces a LogStreamConsumer object that can be used to log messages of severity kVERBOSE
Definition: pluginLogging.h:473
const char * getPluginNamespace() const override
Return the namespace of the plugin creator object.
Definition: embLayerNormVarSeqlenPlugin.cpp:519
Definition: skipLayerNormInt8InterleavedPlugin.h:30
size_t getWorkspaceSize(int) const override
Definition: batchTilePlugin.cpp:86
def test_workspace_size(self, identity_builder_network)
Definition: trt/test_loader.py:185
Severity
Definition: NvInferRuntimeCommon.h:1084
CUresult cuLinkAddData(CUlinkState state, CUjitInputType type, void *data, size_t size, const char *name, unsigned int numOptions, CUjit_option *options, void **optionValues) const
Definition: cudaDriverWrapper.cpp:118
static PluginFieldCollection mFC
Definition: batchTilePlugin.h:110
const char * PYRAMIDROIALGIN_PLUGIN_VERSION
Definition: pyramidROIAlignPlugin.cpp:28
def test_setitem(self, run_results)
Definition: comparator/test_struct.py:61
const char * getPluginVersion() const override
Return the plugin version.
Definition: batchedNMSPlugin.cpp:501
std::array< std::vector< int >, nvinfer1::EnumMax< nvinfer1::OptProfileSelector >()> ShapeRange
Definition: sampleOptions.h:67
def test_layer_with_iterables(self)
Definition: test_ir.py:547
hidden_size
Definition: modules/quant_rnn.py:429
Definition: coordConvACPlugin.h:92
std::vector< std::string > inputSentences
Definition: sampleCharRNN.cpp:131
static expr fmax(float x, float y)
Maximum implementation.
Definition: samples/common/half.h:2569
def test_weight_fake_quant_per_channel(self)
Definition: quant_conv_test.py:307
virtual std::size_t size() const noexcept=0
The size in bytes of the data that was allocated.
def __init__(self, str name, np.dtype dtype=None, Sequence[Union[int, str]] shape=None)
Definition: tensor.py:155
std::string prototxt
Definition: sampleOptions.h:97
void configureWithFormat(const nvinfer1::Dims *inputDims, int nbInputs, const nvinfer1::Dims *outputDims, int nbOutputs, nvinfer1::DataType type, nvinfer1::PluginFormat format, int maxBatchSize) override
Definition: samples/opensource/samplePlugin/fcPlugin.h:119
void serialize(void *buffer) const override
Serialize the layer.
Definition: lReluPlugin.cpp:73
query
Definition: quant_bert.py:64
layers
Definition: demo/Tacotron2/tacotron2/model.py:112
GenerateDetection(int num_classes, int keep_topk, float score_threshold, float iou_threshold, const nvinfer1::Dims &image_size)
Definition: generateDetectionPlugin.cpp:108
def __init__(self, objs)
Definition: Polygraphy/polygraphy/util/misc.py:433
Definition: batchTilePlugin.h:27
parent_obj
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:71
std::string mPrefix
Batch file name prefix.
Definition: BatchStream.h:383
def __init__(self, mode, input_size, hidden_size, num_layers=1, bias=True, batch_first=False, dropout=0, bidirectional=False, **kwargs)
Definition: modules/quant_rnn.py:43
def test_amp_fp16(self)
Definition: integration_test.py:81
nvinfer1::ITensor * recurrentBias
Definition: sampleCharRNN.cpp:253
SampleFasterRCNNParams mParams
The parameters for the sample.
Definition: sampleFasterRCNN.cpp:83
def convert_ids_to_tokens(inv_vocab, ids)
Definition: tokenization.py:147
enable< std::basic_ostream< charT, traits > &, T >::type operator<<(std::basic_ostream< charT, traits > &out, T arg)
Output operator.
Definition: ieee_half.h:2211
~GridAnchorGenerator() override
Definition: gridAnchorPlugin.cpp:160
def __init__(self, shape, dtype=trt.int32)
Definition: perf.py:29
SampleSSD(const SampleSSDParams ¶ms)
Definition: sampleSSD.cpp:66
LogStreamConsumerBuffer mBuffer
Definition: pluginLogging.h:117
void attachToContext(cudnnContext *, cublasContext *, nvinfer1::IGpuAllocator *) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: split.h:128
const char * getPluginVersion() const override
Return the plugin version.
Definition: regionPlugin.cpp:484
int32_t nbOutputs
Definition: sampleAlgorithmSelector.cpp:240
_num_bits
Definition: tools/pytorch-quantization/pytorch_quantization/calib/calibrator.py:32
def test_register_opset(self)
Definition: test_ir.py:473
SampleUffSSD(const SampleUffSSDParams ¶ms)
Definition: sampleUffSSD.cpp:69
HostMemory(std::size_t size, DataType type)
Definition: samples/common/common.h:324
void addToModel(nvinfer1::INetworkDefinition *network, nvinfer1::ITensor *actualInputSequenceLengths, nvinfer1::ITensor *memoryStates, nvinfer1::ITensor *alignmentScores, nvinfer1::ITensor **contextOutput)
add the context vector calculation to the network
Definition: contextNMT.cpp:24
bool mPrintLayercInfo
Definition: sampleConfig.h:54
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: sampleUffPluginV2Ext.cpp:568
BuilderFlag
List of valid modes that the builder can enable when creating an engine from a network definition.
Definition: NvInfer.h:6384
int dataSize
Definition: sampleCharRNN.cpp:119
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: proposalLayerPlugin.cpp:330
const char * NMS_PLUGIN_NAME
Definition: nmsPlugin.cpp:31
int scalarPerVector
Definition: sampleReformatFreeIO.cpp:119
std::string fileName
Definition: samples/common/common.h:662
static int reportFail(const TestAtom &testAtom)
Definition: pluginLogging.h:374
std::vector< float > mFileLabels
List of label files.
Definition: BatchStream.h:382
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: resizeNearestPlugin.cpp:88
static const float MASK_THRESHOLD
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:33
indent
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:30
CublasError(const char *fl, const char *fn, int ln, int stat, const char *msg=nullptr)
Definition: checkMacrosPlugin.h:122
def __call__(self)
Definition: backend/trt/loader.py:153
std::shared_ptr< DeviceBuffer< T > > ptr
Definition: deviceBuffer.h:30
QuantDescriptor
Definition: tensor_quant.py:222
void * bnBias
Definition: groupNormalizationPlugin.h:106
int32_t topK
Definition: NvInferPluginUtils.h:250
unsigned int fused_multihead_attention_int8_384_64_kernel_sm80_cu_o_len
Definition: fused_multihead_attention_int8_384_64_kernel.sm80.cpp:2721
const char * getPluginType() const override
Return the plugin type.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:209
def test_validate_inf(self)
Definition: test_comparator.py:139
def test_input_fake_quant(self)
Definition: quant_pooling_test.py:117
virtual int getEndSequenceId()=0
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: generateDetectionPlugin.cpp:175
def test_k_can_exceed_array_len(self)
Definition: test_postprocess.py:28
The SampleUffSSDParams structure groups the additional parameters required by the Uff SSD sample.
Definition: sampleUffSSD.cpp:48
int initialize() override
Initialize the layer for execution.
Definition: detectionLayerPlugin.cpp:123
A network definition for input to the builder.
Definition: NvInfer.h:4907
def _tokenize_chinese_chars(self, text)
Definition: tokenization.py:281
std::vector< float > _h_scale
Definition: instanceNormalizationPlugin.h:92
Layer that represents a reduction operator across Shape, Int32, Float, and Half tensors.
Definition: NvInfer.h:3579
The TensorRT UFF parser API namespace.
head_size
Definition: builder.py:96
~GenerateDetectionPluginCreator()
Definition: generateDetectionPlugin.h:117
nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType *inputType, int nbInputs) const override
Definition: batchedNMSPlugin.cpp:429
Definition: comparator/struct.py:1
def test_extend_0_args_1_rv(self)
Definition: test_func.py:46
static const int TOP_DOWN_PYRAMID_SIZE
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:57
std::string mFullTextFilename
Definition: sampleConfig.h:39
linear
Definition: print_test.py:49
static const char * testResultString(TestResult result)
returns an appropriate string for prefixing a test result message with the given result
Definition: pluginLogging.h:416
int classes
Definition: regionPlugin.h:95
@ kPOW
The first element to the power of the second element.
float * _d_bias
Definition: instanceNormalizationPlugin.h:95
def deactivate_impl(self)
Definition: backend/base/runner.py:135
int has_arg
Definition: getopt.h:77
def test_layer_with_tensors(self)
Definition: test_ir.py:504
float raw[MaskRCNNConfig::MASK_POOL_SIZE *2 *MaskRCNNConfig::MASK_POOL_SIZE *2]
Definition: sampleUffMaskRCNN.cpp:59
def test_empty_tensor_to_host(self)
Definition: test_cuda.py:88
static half frexp(half arg, int *exp)
Decompression implementation.
Definition: samples/common/half.h:2153
type
Definition: dumpTFWts.py:35
def printable_text(text)
Definition: tokenization.py:97
virtual int getMemoryStatesSize()=0
get the size of the memory state vector
Definition: binaryProtoBlob.h:26
Dims2()
Construct an empty Dims2 object.
Definition: NvInfer.h:71
_num_bits
Definition: tensor_quant.py:79
detail::enable< half &, T >::type operator/=(T rhs)
Arithmetic assignment.
Definition: samples/common/half.h:1471
~GridAnchorPluginCreator() override=default
constexpr int32_t EnumMax< TensorFormat >()
Maximum number of elements in TensorFormat enum.
Definition: NvInferRuntimeCommon.h:352
virtual ~IParametricReLULayer() noexcept
Definition: NvInfer.h:4323
def test_no_shape_tensor_false_positive_negative_dims(self)
Definition: test_data_loader.py:62
filter_length
Definition: stft.py:46
void setLayerPrecision(SampleUniquePtr< nvinfer1::INetworkDefinition > &network)
Sets computation precision for network layers.
Definition: sampleINT8API.cpp:211
attention_head_size
Definition: quant_bert.py:60
CalibrationAlgoType getAlgorithm() override
Signal that this is the entropy calibrator.
Definition: NvInfer.h:6069
Weights minSize
Definition: priorBoxPlugin.h:94
virtual void configurePlugin(const DynamicPluginTensorDesc *in, int32_t nbInputs, const DynamicPluginTensorDesc *out, int32_t nbOutputs)=0
Configure the layer.
Definition: onnxrt/test_runner.py:24
def __init__(self, logger, severity)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:42
const char * getPluginName() const override
Return the plugin name.
Definition: groupNormalizationPlugin.cpp:260
const FusedMultiHeadAttentionXMMAKernel * getXMMAKernels(Data_type type, unsigned int sm)
Definition: fused_multihead_attention.h:334
samplesCommon::CaffeSampleParams mParams
Definition: sampleGoogleNet.cpp:72
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: reorgPlugin.cpp:142
xy_t mFeatureSpatialSize[mFeatureMapCount]
Definition: pyramidROIAlignPlugin.h:100
def suppress(self, severity=CRITICAL)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:142
const void * values
The weight values, in a contiguous array.
Definition: NvInferRuntime.h:81
float mScoreThreshold
Definition: generateDetectionPlugin.h:97
float milliseconds() const noexcept
Definition: samples/common/common.h:790
static long lround(half arg)
Nearest integer implementation.
Definition: samples/common/half.h:2110
const char * RPROI_PLUGIN_VERSION
Definition: nvFasterRCNNPlugin.cpp:30
Dims dims
Definition: NvInferRuntimeCommon.h:369
std::vector< std::pair< int32_t, float > > itemProbPairVec
Definition: sampleMovieLens.cpp:55
Definition: mrcnn_to_trt_single.py:75
samplesCommon::OnnxSampleParams mParams
The parameters for the sample.
Definition: sampleOnnxMnistCoordConvAC.cpp:75
std::string mTextFilename
Definition: sampleConfig.h:38
static PluginFieldCollection mFC
Definition: proposalPlugin.h:143
Bbox(T xmin, T ymin, T xmax, T ymax)
Definition: bboxUtils.h:29
static half cast_impl(U arg, false_type)
Definition: samples/common/half.h:2632
bool force_unroll
Definition: fused_multihead_attention_v2.h:70
Builds an engine from a network definition.
Definition: NvInfer.h:6928
def test_no_quant_all_modes(self, verbose)
Definition: quant_rnn_test.py:338
static half floor(half arg)
Floor implementation.
Definition: samples/common/half.h:2078
void serialize(void *buffer) const override
Serialize the layer.
Definition: pyramidROIAlignPlugin.cpp:206
bool getBatch(void *bindings[], const char *names[], int nbBindings) override
Definition: sampleEngines.cpp:222
void readDataFile(const std::string &dataFilePath)
Definition: BatchStream.h:99
half nextafter(half from, half to)
Next representable value.
Definition: ieee_half.h:2684
void setPluginNamespace(const char *libNamespace) override
Set the namespace that this plugin object belongs to.
Definition: specialSlicePlugin.cpp:94
static expr exp2(float arg)
Binary exponential implementation.
Definition: samples/common/half.h:1787
@ kINT8
8-bit integer representing a quantized floating-point value.
void reset() noexcept
Definition: samples/common/common.h:798
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: priorBoxPlugin.cpp:143
half fabs(half arg)
Absolute value.
Definition: ieee_half.h:2235
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleUffPluginV2Ext.cpp:265
size_t mInputCount
Definition: EntropyCalibrator.h:88
std::string weightsFile
The filename of the weights file.
Definition: sampleMNISTAPI.cpp:51
n_channels
Definition: demo/Tacotron2/waveglow/model.py:94
virtual ~IDeconvolutionLayer()
Definition: NvInfer.h:2436
int mOutputChannelCount
Definition: slpAttention.h:50
@ kDLA_HWC4
DLA image format.
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: cropAndResizePlugin.cpp:189
virtual Dims getBindingDimensions(int32_t bindingIndex) const noexcept=0
Get the dimensions of a binding.
bool constructNetwork(SampleUniquePtr< nvcaffeparser1::ICaffeParser > &parser, SampleUniquePtr< nvinfer1::INetworkDefinition > &network)
uses a Caffe parser to create the MNIST Network and marks the output layers
Definition: sampleMNIST.cpp:240
def get_parser()
Definition: classification_flow.py:54
def __init__(self, bert_config_path, use_fp16, use_int8, use_qat, interleaved)
Definition: builder_varseqlen.py:90
virtual float * getLabels()=0
void processTimestep(int validSampleCount, const float *hCombinedLikelihoods, const int *hVocabularyIndices, const int *hRayOptionIndices, int *hSourceRayIndices, float *hSourceLikelihoods)
Definition: beamSearchPolicy.cpp:59
CUresult(* _cuFuncSetAttribute)(CUfunction, CUfunction_attribute, int)
Definition: cudaDriverWrapper.h:71
virtual nvinfer1::DataType getModelDtype() const =0
Get the Model Data Type.
const char * getPluginVersion() const override
Return the plugin version.
Definition: split.h:106
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: groupNormalizationPlugin.cpp:228
def __init__(self, encoder_n_convolutions, encoder_embedding_dim, encoder_kernel_size)
Definition: demo/Tacotron2/tacotron2/model.py:179
~GridAnchorRectPluginCreator() override=default
output
Definition: 05_folding_constants/generate.py:36
The GenericBuffer class is a templated class for buffers.
Definition: buffers.h:52
bool processInput(const samplesCommon::BufferManager &buffers)
Definition: sampleUffMaskRCNN.cpp:481
@ kINT32
INT32 field type.
virtual bool hasImplicitBatchDimension() const =0
Query whether the engine was built with an implicit batch dimension.
def __repr__(self)
Definition: node.py:112
virtual bool isPlugin(const char *layerName)=0
A user implemented function that determines if a layer configuration is provided by an IPlugin.
def test_fake_quant_per_tensor(self)
Definition: quant_conv_test.py:129
def __call__(self)
Definition: backend/tf/loader.py:409
def __call__(self)
Definition: backend/trt/loader.py:104
const trtcaffe::BlobProto * getBlob(const std::string &layerName, int index)
Definition: caffeWeightFactory.cpp:105
float __half2float(__half h)
Definition: fp16.h:86
virtual int32_t getNbOutputs() const =0
Get the number of outputs in the network.
linear_projection
Definition: export_tacotron2_onnx.py:128
nvinfer1::IPluginV2DynamicExt * deserializePlugin(const char *, const void *serialData, size_t serialLength)
Called during deserialization of plugin layer.
Definition: split.h:164
bool build()
Builds the network engine.
Definition: sampleUffMNIST.cpp:115
def binding(self)
Definition: perf.py:32
bool verifyOutput(const samplesCommon::BufferManager &buffers)
Definition: sampleUffMaskRCNN.cpp:578
cudaStream_t mStream
Definition: sampleDevice.h:100
@ kINT16
INT16 field type.
Definition: backend/onnxrt/runner.py:1
def __repr__(self)
Definition: backend/trt/loader.py:254
def transliteration_cleaners(text)
Definition: cleaners.py:90
def adjust_learning_rate(iteration, epoch, optimizer, learning_rate, anneal_steps, anneal_factor, rank)
Definition: train.py:305
Definition: backend/trt/loader.py:140
~Batch()
Definition: sampleMovieLensMPS.cpp:236
def check_file_non_empty(path)
Definition: common.py:23
cudnnTensorDescriptor_t _b_desc
Definition: instanceNormalizationPlugin.h:98
An array of weights used as a layer parameter.
Definition: NvInferRuntime.h:77
def __exit__(self, exc_type, exc_value, traceback)
Definition: backend/base/runner.py:71
@ kHWC8
Eight channel format where C is padded to a multiple of 8.
const IBlobNameToTensor * parse(const char *deploy, const char *model, nvinfer1::INetworkDefinition &network, nvinfer1::DataType weightType) override
Parse a prototxt file and a binaryproto Caffe model to extract network definition and weights associa...
Definition: caffeParser.cpp:341
InferenceTime traceToTiming(const InferenceTrace &a)
Definition: sampleReporting.cpp:71
def __repr__(self)
Definition: histogram.py:251
nvinfer1::DimsHW preRefineBboxDims
Definition: maskRCNNKernels.h:128
LogStreamConsumer gLogInfo
Definition: logger.cpp:24
def parse_args(parser)
Definition: main.py:21
enable< expr, T, U >::type operator-(T x, U y)
Subtract halfs.
Definition: ieee_half.h:2178
def main()
Definition: profiler.py:157
def __init__(self, quant_desc=QuantDescriptor(), disabled=False, if_quant=True, if_clip=False, if_calib=False)
Definition: tensor_quantizer.py:70
def main()
Definition: preprocess_audio2mel.py:70
def _is_chinese_char(self, cp)
Definition: tokenization.py:294
def printHeader(allFeatures, selection, gp=False, count=False)
Definition: prn_utils.py:42
Definition: backend/onnx/loader.py:107
fp16
Definition: backend/trt_legacy.py:217
def test_can_name_runner(self)
Definition: onnx/test_runner.py:22
Convenience object used to facilitate use of C++ stream syntax when logging messages....
Definition: pluginLogging.h:129
constexpr int32_t EnumMax< DataType >()
Maximum number of elements in DataType enum.
Definition: NvInferRuntimeCommon.h:156
A deconvolution layer in a network definition.
Definition: NvInfer.h:2283
static bool isinf(half arg)
Classification implementation.
Definition: samples/common/half.h:2351
The TensorRT API version 1 namespace.
samplesCommon::UffSampleParams mParams
Definition: sampleUffMNIST.cpp:101
InferenceTime & operator=(const InferenceTime &)=default
nvinfer1::DimsExprs getOutputDimensions(int outputIndex, const nvinfer1::DimsExprs *inputs, int nbInputs, nvinfer1::IExprBuilder &exprBuilder) override
size_t getWorkspaceSize(int) const override
Definition: resizeNearestPlugin.cpp:126
virtual bool supportsModel(void const *serialized_onnx_model, size_t serialized_onnx_model_size, SubGraphCollection_t &sub_graph_collection)=0
Check whether TensorRT supports a particular ONNX model.
doc_stride
Definition: demo/BERT/helpers/calibrator.py:42
const char * HIDDEN_IN_BLOB_NAME
Definition: sampleCharRNN.cpp:88
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: proposalPlugin.cpp:315
virtual const char * getPluginNamespace() const =0
Return the namespace of the plugin creator object.
virtual bool supportsFormatCombination(int32_t pos, const PluginTensorDesc *inOut, int32_t nbInputs, int32_t nbOutputs) const =0
Return true if plugin supports the format and datatype for the input/output indexed by pos.
def test_loader(self)
Definition: trt/test_loader.py:90
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:41
CUresult(* _cuLinkAddData)(CUlinkState, CUjitInputType, void *, size_t, const char *, unsigned int, CUjit_option *, void **)
Definition: cudaDriverWrapper.h:79
encodes single input into output states
Definition: decoder.h:33
Definition: onnx/test_loader.py:74
def test_insert(self, field_names)
Definition: test_ir.py:245
Definition: quant_pooling_test.py:197
optimize
Definition: backend/onnx/loader.py:70
int mNbClasses
Definition: detectionLayerPlugin.h:94
TypedHostMemory(std::size_t size)
Definition: samples/common/common.h:338
Definition: demo/Tacotron2/tacotron2/model.py:174
bool teardown()
Cleans up any state created in the sample class.
Definition: sampleUffSSD.cpp:274
bool isgreater(half x, half y)
Comparison for greater than.
Definition: ieee_half.h:2773
constexpr int32_t EnumMax< MatrixOperation >()
Maximum number of elements in MatrixOperation enum.
Definition: NvInfer.h:4156
minimum_segment_size
Definition: backend/tf/loader.py:231
_sess
Definition: backend/tf/runner.py:51
def setup_method(self)
Definition: test_ir.py:135
def __init__(self, sigma=1.0)
Definition: waveglow/loss_function.py:20
void add(const std::string &token)
add new token to vocabulary, ID is auto-generated
Definition: vocabulary.cpp:34
W
Definition: onnx-graphsurgeon/examples/02_creating_a_model_with_initializer/example.py:24
std::vector< errorPair > errorStack
Definition: ErrorRecorder.h:40
SamplePlugin(const samplesCommon::CaffeSampleParams ¶ms)
Definition: samplePlugin.cpp:61
buf
Definition: perf_varseqlen.py:30
BatchStream(int batchSize, int maxBatches, const nvinfer1::Dims &dims, const std::string &listFile, const std::vector< std::string > &directories)
Definition: frcnnUtils.h:227
def __init__(self, int num_features, float eps=1e-5, float momentum=0.1, bool affine=False, bool track_running_stats=False, **kwargs)
Definition: quant_instancenorm.py:68
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: batchTilePlugin.cpp:147
void helpHelp(std::ostream &os)
Definition: sampleOptions.cpp:1300
__host__ __device__ int div_up(int a, int b)
Definition: reducedMathPlugin.h:47
int initialize() override
Initialize the layer for execution.
Definition: samples/opensource/samplePlugin/fcPlugin.h:127
virtual void configure(const Dims *inputDims, int32_t nbInputs, const Dims *outputDims, int32_t nbOutputs, int32_t maxBatchSize)=0
Configure the layer.
void destroy() override
Definition: binaryProtoBlob.h:51
The SampleDynamicReshape class implementes the dynamic reshape sample.
Definition: sampleDynamicReshape.cpp:45
def printCsv(data, count=False)
Definition: prn_utils.py:61
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: detectionLayerPlugin.cpp:307
Definition: samples/common/common.h:660
prenet
Definition: demo/Tacotron2/tacotron2/model.py:254
virtual void start()
Definition: samples/common/common.h:784
nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: skipLayerNormInt8InterleavedPlugin.cpp:201
BatchStream(int batchSize, int maxBatches, const std::string &prefix, const std::string &suffix, const std::vector< std::string > &directories)
Definition: frcnnUtils.h:194
static int reportWaive(const TestAtom &testAtom)
Definition: samples/common/logging.h:359
void serialize(void *buffer) const override
Serialize the layer.
Definition: resizeNearestPlugin.cpp:137
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: coordConvACPlugin.cpp:124
virtual bool hasImplicitBatchDimension() const =0
Query whether the network was created with an implicit batch dimension.
bool processInput(const samplesCommon::BufferManager &buffers)
Reads the input and stores the result in a managed buffer.
Definition: sampleMNISTAPI.cpp:304
void destroy() override
Destroy the plugin object.
Definition: proposalLayerPlugin.cpp:155
bool flip
Definition: NvInferPluginUtils.h:95
float_range
Definition: comparator/data_loader.py:90
const char * SPECIALSLICE_PLUGIN_VERSION
Definition: specialSlicePlugin.cpp:28
static T cast_impl(half arg, false_type)
Definition: samples/common/half.h:2654
def to_constant(self, np.ndarray values)
Definition: tensor.py:171
virtual ~IBinaryProtoBlob()
Definition: NvCaffeParser.h:80
def test_per_channel_scale(self, verbose)
Definition: tensor_quantizer_test.py:67
__host__ __device__ __forceinline__ int mod(int x) const
Definition: reducedMathPlugin.h:76
int mMaxBatches
Definition: BatchStream.h:374
float findPercentile(float percentage, const std::vector< InferenceTime > &timings, const T &toFloat)
Find percentile in an ascending sequence of timings.
Definition: sampleReporting.cpp:40
PinnedHostBuffer< float >::ptr mData
Definition: debugUtil.h:72
samplesCommon::UffSampleParams mParams
Definition: sampleUffPluginV2Ext.cpp:334
Definition: tensor_quant.py:300
def test_handle_is_ctypes_ptr(self)
Definition: test_cuda.py:102
def forward(self, x)
Definition: layers.py:32
RPROIPlugin(RPROIParams params, const float *anchorsRatios, const float *anchorsScales)
Definition: nvFasterRCNNPlugin.cpp:37
def test_backward(self)
Definition: tensor_quant_test.py:84
def __init__(self, sess, name=None)
Definition: backend/onnxrt/runner.py:29
def test_input_fake_quant(self)
Definition: quant_pooling_test.py:69
Layer that represents a constant value.
Definition: NvInfer.h:4270
static expr tgamma(float arg)
Gamma implementation.
Definition: samples/common/half.h:2052
Definition: sampleOptions.h:186
def parse_args(parser)
Definition: inference_perf.py:31
size_t getSerializationSize() override
Find the size of the serialization buffer required.
Definition: debugUtil.cpp:112
int w
Definition: sampleUffMaskRCNN.cpp:75
CalibrationAlgoType getAlgorithm() override
Signal that this is the legacy calibrator.
Definition: NvInfer.h:6114
unsigned int fused_multihead_attention_v2_fp16_256_64_kernel_sm75_cubin_len
Definition: fused_multihead_attention_v2_fp16_256_64_kernel.sm75.cpp:5789
void run(const void *inputIds, const void *segmentIds, const void *inputMask, int warmUps, int iterations)
Definition: bert_infer.h:283
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: embLayerNormVarSeqlenPlugin.cpp:362
int32_t
Definition: NvInferRuntimeCommon.h:792
IPluginV2Ext * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: gridAnchorPlugin.cpp:487
const char * getPluginName() const override
Return the plugin name.
Definition: gridAnchorPlugin.cpp:364
v
Definition: demo/Tacotron2/tacotron2/model.py:55
int mNumInputs
Definition: slpEmbedder.h:53
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace of the plugin creator based on the plugin library it belongs to.
Definition: embLayerNormVarSeqlenPlugin.cpp:514
def test_cli_examples(example)
Definition: Polygraphy/tests/test_examples.py:118
virtual ~IHostMemory()
Definition: NvInferRuntime.h:103
Weights maxSize
Definition: priorBoxPlugin.h:94
static half nextafter(half from, half to)
Enumeration implementation.
Definition: samples/common/half.h:2287
detail::enable< half &, T >::type operator-=(T rhs)
Arithmetic assignment.
Definition: samples/common/half.h:1451
void destroy() override
Destroy the plugin object.
Definition: generateDetectionPlugin.cpp:155
int boxesSize
Definition: batchedNMSPlugin.h:113
cudaGraphExec_t mGraphExec
Definition: sampleDevice.h:206
const char * REGION_PLUGIN_NAME
Definition: regionPlugin.cpp:28
Application-implemented interface for calibration.
Definition: NvInfer.h:5998
size_t totalSize
Definition: maskRCNNKernels.h:152
ErrorCode
Definition: NvOnnxParser.h:66
def test_fake_quant_per_channel(self)
Definition: quant_instancenorm_test.py:130
std::vector< std::shared_ptr< CudaBind< float > > > mAnchorBoxesDevice
Definition: multilevelProposeROIPlugin.h:105
void serialize(void *buffer) const override
Serialize the layer.
Definition: groupNormalizationPlugin.cpp:167
file
Definition: evaluate-v1.1.py:110
float maskThreshold
Definition: sampleUffMaskRCNN.cpp:307
virtual IErrorRecorder * getErrorRecorder() const noexcept=0
get the ErrorRecorder assigned to this interface.
int mAnchorsCnt
Definition: generateDetectionPlugin.h:101
def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0, output_padding=0, groups=1, bias=True, dilation=1, padding_mode='zeros', **kwargs)
Definition: quant_conv.py:274
bool build(DataType dataType)
Function builds the network engine.
Definition: sampleINT8.cpp:127
std::pair< nvinfer1::DataType, nvinfer1::TensorFormats > IOFormat
Definition: sampleOptions.h:65
expr atan2(half x, half y)
Arc tangent function.
Definition: ieee_half.h:2460
def test_per_output_tol(self, mode)
Definition: test_compare.py:34
IUffParser * createUffParser()
Creates a IUffParser object.
@ kDEFAULT
Full capability, TensorRT mode without any restrictions using TensorRT nvinfer1 APIs.
bool supportsFormat(nvinfer1::DataType type, nvinfer1::PluginFormat format) const override
Check format support.
Definition: samples/opensource/samplePlugin/fcPlugin.h:107
int64_t count
The number of weights in the array.
Definition: NvInferRuntime.h:82
def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros', **kwargs)
Definition: quant_conv.py:183
@ kFLOOR_DIV
Floor division of the first element by the second.
SampleUniquePtr< nvcaffeparser1::IBinaryProtoBlob > mMeanBlob
The mean blob, which need to keep around until build time.
Definition: samplePlugin.cpp:112
virtual nvinfer1::ICudaEngine * buildEngineWithConfig(INetworkDefinition &network, IBuilderConfig &config)=0
Builds an engine for the given INetworkDefinition and given IBuilderConfig.
std::string magic
Definition: sampleUffMaskRCNN.cpp:74
@ kWAIVED
The test was waived.
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: coordConvACPlugin.cpp:71
def _should_keep_symbol(s)
Definition: demo/Tacotron2/tacotron2/text/__init__.py:73
def load_graph(path)
Definition: tf/util.py:23
inputs
Definition: tensor.py:165
~PriorBoxPluginCreator() override
Definition: priorBoxPlugin.cpp:363
float mThresh
Definition: multilevelCropAndResizePlugin.h:98
ComponentWeights::ptr mWeights
Definition: multiplicativeAlignment.h:51
void serialize(void *buffer) const override
Serialize the layer.
Definition: nvFasterRCNNPlugin.cpp:182
def num_bits(self)
Definition: tensor_quant.py:112
def unwrap_distributed(state_dict)
Definition: Tacotron2/inference.py:86
def convert_conv_1d_to_2d(conv1d)
Definition: export_waveglow_onnx.py:54
bool overlap
Definition: sampleOptions.h:154
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: instanceNormalizationPlugin.cpp:345
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: resizeNearestPlugin.cpp:175
def try_send_on_queue(queue, obj)
Definition: Polygraphy/polygraphy/util/misc.py:227
static const int POOL_SIZE
Definition: tlt_mrcnn_config.h:30
static std::vector< PluginField > mPluginAttributes
Definition: coordConvACPlugin.h:111
static bool areSame(const AlgorithmCacheItem &algoCacheItem, const IAlgorithm &algoChoice)
The combination of implementation, tactic and input/output formats is unique to an algorithm,...
Definition: sampleAlgorithmSelector.cpp:247
def cleanup(self, remove_unused_node_outputs=False)
Definition: graph.py:198
DetectionOutputParameters param
Definition: nmsPlugin.h:89
An engine for executing inference on a built network, with functionally unsafe features.
Definition: NvInferRuntime.h:1157
def tensors_linear_graph()
Definition: test_ir.py:319
@ kHOST
Data stored on host.
virtual ~IConstantLayer()
Definition: NvInfer.h:4310
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: lReluPlugin.cpp:67
def test_import_graph_value_info(self)
Definition: test_importers.py:98
def test_can_construct_onnx_loader()
Definition: test_tensorrt_legacy.py:37
std::string mNameSpace
Definition: multilevelProposeROIPlugin.h:117
def verbose(request)
Definition: tools/pytorch-quantization/tests/fixtures/__init__.py:20
new
Definition: test_cuda.py:29
bool mShouldLog
Definition: pluginLogging.h:100
def update(filename, args)
Definition: copyright-scan.py:109
def __call__(self)
Definition: backend/tf/loader.py:235
_axis
Definition: tensor_quantizer.py:76
const char * getPluginType() const override
Return the plugin type.
Definition: flattenConcat.cpp:269
def mark_outputs(network, outputs)
Definition: trt/util.py:197
def extra_repr(self)
Definition: modules/quant_rnn.py:310
virtual void setFlag(BuilderFlag builderFlag)=0
Set a single build mode flag.
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: nmsPlugin.cpp:299
bool builderCache
Definition: sampleOptions.h:119
int mKeepTopK
Definition: multilevelProposeROIPlugin.h:96
nvinfer1::ITensor * data
Definition: sampleCharRNN.cpp:243
nvinfer1::DataType mType
Definition: generateDetectionPlugin.h:103
std::string mPluginNamespace
Definition: nmsPlugin.h:91
int mOutputChannelCount
Definition: multiplicativeAlignment.h:54
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: groupNormalizationPlugin.cpp:181
@ kATANH
Inverse hyperbolic tangent.
std::vector< char > mCalibrationCache
Definition: EntropyCalibrator.h:93
bool isOK()
Definition: blobNameToTensor.h:61
const char * getPluginType() const override
Return the plugin type.
Definition: regionPlugin.cpp:383
std::string mCalibrationFilename
Definition: sampleConfig.h:43
bool supportsFormatCombination(int pos, const nvinfer1::PluginTensorDesc *inOut, int nbInputs, int nbOutputs) override
Definition: embLayerNormVarSeqlenPlugin.cpp:174
SampleUffFasterRcnn(const SampleUffFasterRcnnParams ¶ms)
Definition: sampleUffFasterRCNN.cpp:99
std::vector< samplesCommon::PPM< 3, 300, 300 > > mPPMs
PPMs of test images.
Definition: sampleSSD.cpp:92
half type
Definition: ieee_half.h:394
def test_equals_name_mismatch(self)
Definition: test_ir.py:143
_model
Definition: backend/pyt/runner.py:38
score_mask_value
Definition: demo/Tacotron2/tacotron2/model.py:59
Definition: backend/onnx/loader.py:24
def __str__(self)
Definition: Polygraphy/tests/test_examples.py:87
SampleSSDParams mParams
The parameters for the sample.
Definition: sampleSSD.cpp:88
memory_layer
Definition: demo/Tacotron2/tacotron2/model.py:53
void destroy()
Definition: sampleReformatFreeIO.cpp:174
virtual void log(std::ostream &logStream) const
Definition: checkMacrosPlugin.cpp:112
BatchStream(int batchSize, int maxBatches, nvinfer1::Dims dims, std::string listFile, std::vector< std::string > directories)
Definition: BatchStream.h:191
std::size_t size() const noexcept override
The size in bytes of the data that was allocated.
Definition: samples/common/common.h:314
virtual ~ITensor()
Definition: NvInfer.h:608
int32_t keepTopK
Definition: NvInferPluginUtils.h:184
def write_histogram_cache(self, ptr, length)
Definition: demo/BERT/helpers/calibrator.py:110
def __init__(self, do_lower_case=True)
Definition: tokenization.py:218
inference_time
Definition: backend/base/runner.py:45
Definition: NvInferRuntime.h:387
def batchnorm_to_float(module)
Definition: demo/Tacotron2/models.py:37
def error(self, message, mode=LogMode.EACH)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:239
std::vector< std::pair< int, std::vector< std::string > > > values
Definition: getOptions.h:60
std::vector< std::string > dataDirs
Definition: sampleINT8API.cpp:63
int minTiming
Definition: sampleOptions.h:110
~BenchmarkWriter() override=default
int32_t backgroundLabelId
Definition: NvInferPluginUtils.h:250
Definition: backend/base/runner.py:23
void wait(EventType e, StreamType s)
Definition: sampleInference.cpp:413
QKVToContextInterleavedPlugin()=delete
def load_onnx_weights_and_quant(path, config)
Definition: builder_varseqlen.py:456
_all_weights
Definition: modules/quant_rnn.py:74
name
Definition: backend/base/runner.py:44
virtual float getDynamicRangeMax(const char *tensorName) const =0
Get maximum of dynamic range.
Definition: base_importer.py:1
float prob
Definition: sampleUffMaskRCNN.cpp:66
def tokenize(self, text)
Definition: tokenization.py:194
T half2int_up(uint16 value)
Convert half-precision floating point to integer using round-to-nearest-away-from-zero.
Definition: ieee_half.h:1001
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: generateDetectionPlugin.cpp:185
def parse_args()
Definition: inference_c.py:39
all it does is to measure the performance of sequence generation
Definition: benchmarkWriter.h:32
void initialize() override
it is called right before inference starts
Definition: benchmarkWriter.cpp:39
SampleBuffer()
Definition: sampleReformatFreeIO.cpp:136
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: proposalLayerPlugin.cpp:66
def run(self, command)
Definition: Polygraphy/tests/test_examples.py:65
bool verifyOutput(const samplesCommon::BufferManager &buffers, const std::string &outputTensorName, int groundTruthDigit) const
Verifies that the output is correct and prints it.
Definition: samplePlugin.cpp:327
uint32_t scale_softmax
Definition: fused_multihead_attention.h:81
nvinfer1::DataType mType
Definition: embLayerNormVarSeqlenPlugin.h:102
def __exit__(self, exc_type, exc_value, exc_traceback)
Definition: Tacotron2/inference.py:175
def test_two_tensor(self, verbose)
Definition: calibrator_test.py:228
~CropAndResizePluginCreator() override
Definition: cropAndResizePlugin.cpp:251
const char * getPluginVersion() const override
Return the plugin version.
Definition: instanceNormalizationPlugin.cpp:244
std::pair< ErrorCode, std::string > errorPair
Definition: ErrorRecorder.h:39
def test_fake_quant_per_tensor(self)
Definition: quant_linear_test.py:104
nvinfer1::ITensor * addReshape(SampleUniquePtr< nvinfer1::INetworkDefinition > &network, nvinfer1::ITensor &tensor, nvinfer1::Dims dims)
Definition: sampleCharRNN.cpp:558
virtual ~ICaffeParser()
Definition: NvCaffeParser.h:267
Definition: denoiser.py:23
Severity getReportableSeverity() const
Definition: samples/common/logging.h:370
def to_variable(self, np.dtype dtype=None, Sequence[Union[int, str]] shape=[])
Definition: tensor.py:205
virtual const IAlgorithmVariant & getAlgorithmVariant() const =0
Returns the algorithm variant.
explicit_precision
Definition: backend/trt/loader.py:100
model
Definition: backend/cntk/runner.py:28
path
Definition: backend/common/loader.py:27
Definition: sampleDevice.h:297
@ kOPT
This is used to set or get the value that is used in the optimization (kernel selection).
bert::cuda_unique_ptr< void > mPosEmbDev
Definition: embLayerNormVarSeqlenPlugin.h:92
std::string mNamespace
Definition: plugin.h:74
Definition: tensor_quant_test.py:38
Definition: normalizePlugin.h:107
std::string validateTRTOption(const std::set< char > &seenShortNames, const std::set< std::string > &seenLongNames, const TRTOption &opt)
validateTRTOption will return a string containing an error message if options contain non-numeric cha...
Definition: getOptions.cpp:79
bert::WeightsWithOwnership mGamma
Definition: skipLayerNormInt8InterleavedPlugin.h:77
std::string modelFileName
Definition: sampleINT8API.cpp:62
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: nmsPlugin.cpp:257
virtual ~IInt8MinMaxCalibrator()
Definition: NvInfer.h:6101
Definition: BatchStream.h:40
Definition: quant_modules.py:36
void * getHostBuffer() const
Definition: sampleDevice.h:328
const nvinfer1::PluginFieldCollection * getFieldNames()
Return a list of fields that needs to be passed to createPlugin.
Definition: split.h:152
@ kTHRESHOLDED_RELU
Thresholded ReLU activation: x>alpha ? x : 0.
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: batchedNMSPlugin.cpp:210
explicit_batch
Definition: backend/trt/loader.py:72
def load_filepaths_and_text(dataset_path, filename, split="|")
Definition: demo/Tacotron2/common/utils.py:36
void start()
Definition: samples/common/common.h:845
void copyDeviceToInt8Output(const void *src, void *dst)
Definition: sampleUffPluginV2Ext.cpp:613
std::string mnistMeansProto
The proto file containing means.
Definition: sampleMNISTAPI.cpp:52
def __init__(self, severity=INFO, colors=True, letter=True, timestamp=False, line_info=False, exit_on_errors=False)
Definition: Polygraphy/polygraphy/logger/logger.py:115
Definition: tokenization.py:160
def _short_amax(self, fmt='.4f')
Definition: tensor_quantizer.py:347
def test_no_quant(self)
Definition: quant_conv_transposed_test.py:336
@ kIDENTITY
Identity layer.
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: multilevelCropAndResizePlugin.cpp:175
def disable_quant(self)
Definition: tensor_quantizer.py:182
Definition: quant_conv.py:269
@ kERROR
Application error has occurred.
unsigned int fused_multihead_attention_v2_int8_256_64_kernel_sm80_cubin_len
Definition: fused_multihead_attention_v2_int8_256_64_kernel.sm80.cpp:13062
Definition: samples/common/common.h:667
def test_getitem(self, run_results)
Definition: comparator/test_struct.py:46
virtual void destroy()=0
Destroy the plugin object.
int C
Definition: regionPlugin.h:97
def test_shape_tensor_detected(self)
Definition: test_data_loader.py:50
def test_fake_quant_per_channel(self)
Definition: quant_conv_transposed_test.py:142
virtual Dims getOutputDimensions(int32_t index, const Dims *inputs, int32_t nbInputDims)=0
Get the dimension of an output tensor.
static expr acosh(float arg)
Hyperbolic area cosine implementation.
Definition: samples/common/half.h:1981
@ kSTRICT_TYPES
Enables strict type constraints.
encodes input sentences into output states
Definition: encoder.h:33
static const int MASK_POOL_SIZE
Definition: tlt_mrcnn_config.h:31
static half_float::half quiet_NaN()
Quiet NaN.
Definition: samples/common/half.h:4252
Definition: sampleMovieLensMPS.cpp:118
@ kOUTPUT
Output gate (o).
unsigned int fused_multihead_attention_fp16_64_64_kernel_sm80_cu_o_len
Definition: fused_multihead_attention_fp16_64_64_kernel.sm80.cpp:1417
uff_order
Definition: backend/trt_legacy.py:83
int iC
Definition: coordConvACPlugin.h:86
def pickle_save(path, obj)
Definition: Polygraphy/polygraphy/util/misc.py:202
@ kNHWC8
Deprecated name of kHWC8, provided for backwards compatibility and will be removed in TensorRT 8....
x
Definition: 06_removing_nodes/generate.py:23
def get_input_metadata_from_profile(profile, network)
Definition: trt/util.py:339
def backward(ctx, grad_outputs)
Definition: tensor_quant.py:410
DimsExprs getOutputDimensions(int outputIndex, const DimsExprs *inputs, int nbInputs, IExprBuilder &exprBuilder) override
Definition: batchedNMSPlugin.cpp:128
const char * getPluginName() const override
Return the plugin name.
Definition: batchedNMSPlugin.cpp:496
container for static debug utility functions
Definition: debugUtil.h:35
def convert_model(inference_model, output_path, output_nodes=[], preprocessor=None, text=False, list_nodes=False)
Definition: mrcnn_to_trt_single.py:126
void destroy() override
Destroy the plugin object.
Definition: batchTilePlugin.cpp:161
path
Definition: backend/onnx/loader.py:163
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: gridAnchorPlugin.cpp:178
virtual bool parseWithWeightDescriptors(void const *serialized_onnx_model, size_t serialized_onnx_model_size, uint32_t weight_count, onnxTensorDescriptorV1 const *weight_descriptors)=0
Parse a serialized ONNX model into the TensorRT network with consideration of user provided weights.
def is_empty_shape(shape)
Definition: Polygraphy/polygraphy/util/misc.py:168
def __enter__(self)
Definition: Polygraphy/tests/test_examples.py:60
static const size_t kINVALID_SIZE_VALUE
Definition: buffers.h:240
void * mDeviceInput
Definition: EntropyCalibrator.h:92
def forward(self, input)
Definition: quant_pooling.py:112
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: detectionLayerPlugin.cpp:62
@ kDECONVOLUTION
Deconvolution layer.
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleUffMNIST.cpp:99
SampleDynamicReshape(const samplesCommon::OnnxSampleParams ¶ms)
Definition: sampleDynamicReshape.cpp:51
bool strict
Definition: sampleMovieLens.cpp:69
static constexpr int32_t kSM_TURING
Definition: qkvToContextInt8InterleavedPlugin.h:30
std::vector< TrtUniquePtr< nvinfer1::IExecutionContext > > context
Definition: sampleInference.h:37
def Recurrent(inner, reverse=False)
Definition: _functions/quant_rnn.py:130
Definition: demo/Tacotron2/tacotron2/model.py:587
static const std::string MODEL_INPUT
Definition: tlt_mrcnn_config.h:171
def test_non_user_provided_inputs_never_shape_tensors(self)
Definition: test_data_loader.py:88
def __init__(self, input_size, hidden_size, bias=True, **kwargs)
Definition: modules/quant_rnn.py:385
bool shareLocation
Definition: NvInferPluginUtils.h:183
def test_cleanup_independent_path(self)
Definition: test_ir.py:665
A Pooling layer in a network definition.
Definition: NvInfer.h:1731
p_attention_dropout
Definition: export_tacotron2_onnx.py:110
const char * getPluginName() const override
Return the plugin name.
Definition: proposalLayerPlugin.cpp:51
host
Definition: backend/trt_legacy.py:181
def __exit__(self, exc_type, exc_value, traceback)
Definition: Polygraphy/polygraphy/logger/logger.py:68
uint32_t scale_bmm2
Definition: fused_multihead_attention.h:81
unsigned char fused_multihead_attention_v2_fp16_128_64_kernel_sm75_cubin[]
Definition: fused_multihead_attention_v2_fp16_128_64_kernel.sm75.cpp:19
unsigned int mSM
Definition: fused_multihead_attention.h:280
float microseconds() const noexcept
Definition: samples/common/common.h:786
static PluginFieldCollection mFC
Definition: specialSlicePlugin.h:117
_severity
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:100
const char * getPluginName() const override
Return the plugin name.
Definition: coordConvACPlugin.cpp:180
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: regionPlugin.cpp:463
Definition: modules/quant_rnn.py:422
virtual IPluginCreator *const * getPluginCreatorList(int32_t *numCreators) const noexcept=0
Return all the registered plugin creators and the number of registered plugin creators.
Definition: backend/tf/runner.py:1
bool signbit(half arg)
Check sign.
Definition: ieee_half.h:2760
int64_t o_stride_in_bytes
Definition: fused_multihead_attention_v2.h:39
use_int8_multihead
Definition: builder.py:102
static expr expm1(float arg)
Exponential implementation.
Definition: samples/common/half.h:1775
std::string uffFileName
Filename of uff file of a network.
Definition: argsParser.h:73
@ kNORMALIZE
Normalize plugin.
bool constructNetwork(SampleUniquePtr< nvinfer1::IBuilder > &builder, SampleUniquePtr< nvinfer1::INetworkDefinition > &network, SampleUniquePtr< nvuffparser::IUffParser > &parser)
Parses an UFF model for SSD and creates a TensorRT network.
Definition: sampleUffFasterRCNN.cpp:243
Definition: tokenization.py:184
Definition: tools/pytorch-quantization/tests/utils.py:1
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: sampleUffPluginV2Ext.cpp:573
bool processInput(const samplesCommon::BufferManager &buffers)
Reads the input and stores the result in a managed buffer.
Definition: sampleOnnxMnistCoordConvAC.cpp:246
Definition: normalizePlugin.h:31
nvinfer1::ILayer * parseDeconvolution(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &weightFactory, BlobNameToTensor &tensors)
Definition: parseDeconv.cpp:23
def normalize_answer(s)
Definition: evaluate-v1.1.py:29
Dims4()
Construct an empty Dims2 object.
Definition: NvInfer.h:296
@ kUNIDIRECTION
Network iterations from first input to last input.
void operator()(void *ptr)
Definition: sampleDevice.h:283
def activate_impl(self)
Definition: backend/tf/runner.py:64
virtual void destroy() noexcept=0
Destroy the allocated memory.
BertInference bert
Definition: infer_c.cpp:50
Definition: batchedNMSPlugin.h:149
std::unordered_set< std::string > mRegistryList
Definition: InferPlugin.cpp:144
std::vector< nvinfer1::Weights > mGateBiasWeights
Definition: lstmEncoder.h:52
std::vector< std::string > inputTensorNames
Definition: argsParser.h:43
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: multilevelProposeROIPlugin.cpp:472
virtual int getRefitMap(const char **weightNames, const char **layerNames, nvinfer1::WeightsRole *roles)=0
Get description of all ONNX weights that can be refitted.
int mOutputConcatAxis
Definition: flattenConcat.h:102
Wrapper implementing unspecialized half-precision functions.
Definition: ieee_half.h:1215
void reportAlgorithms(const nvinfer1::IAlgorithmContext *const *algoContexts, const nvinfer1::IAlgorithm *const *algoChoices, int32_t nbAlgorithms) override
Called by TensorRT to report choices it made.
Definition: sampleAlgorithmSelector.cpp:300
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: reorgPlugin.cpp:205
def forward(self, x)
Definition: demo/Tacotron2/tacotron2/model.py:162
nvinfer1::DimsHW sortClassPosDims
Definition: maskRCNNKernels.h:104
Count_t ngramCounts(const Segment_t &segment, int maxOrder=4)
Definition: bleuScoreWriter.cpp:70
Definition: waveglow/data_function.py:23
int b
Definition: fused_multihead_attention.h:79
@ kACOSH
Inverse hyperbolic cosine.
SampleMNISTAPI(const SampleMNISTAPIParams ¶ms)
Definition: sampleMNISTAPI.cpp:65
A concatenation layer in a network definition.
Definition: NvInfer.h:2248
std::shared_ptr< DataWriter > ptr
Definition: dataWriter.h:36
FCPlugin(const nvinfer1::Weights *weights, int nbWeights, int nbOutputChannels)
Definition: samples/opensource/samplePlugin/fcPlugin.h:39
def test_backward(self)
Definition: integration_test.py:69
def reduce_prod(self, a, axes, keepdims=True)
Definition: 09_shape_operations_with_the_layer_api/generate.py:31
~GroupNormalizationPluginCreator() override=default
tf32
Definition: backend/trt_legacy.py:218
bool writeNetworkTensors
Definition: sampleINT8API.cpp:650
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: priorBoxPlugin.cpp:262
LogStreamConsumer LOG_ERROR(const Logger &logger)
produces a LogStreamConsumer object that can be used to log messages of severity kERROR
Definition: pluginLogging.h:509
std::vector< void * > & getDeviceBindings()
Returns a vector of device buffers that you can use directly as bindings for the execute and enqueue ...
Definition: buffers.h:278
T type
Definition: ieee_half.h:377
float score
Definition: sampleUffMaskRCNN.cpp:54
void dumpInputs(const nvinfer1::IExecutionContext &context, const Bindings &bindings, std::ostream &os)
Print input tensors to stream.
Definition: sampleReporting.cpp:323
dropout
Definition: quant_bert.py:68
void destroy() override
Destroy the plugin object.
Definition: groupNormalizationPlugin.cpp:187
def infer(self, z)
Definition: demo/Tacotron2/waveglow/model.py:65
cudaEvent_t mStop
Definition: samples/common/common.h:835
virtual bool registerCreator(IPluginCreator &creator, const char *pluginNamespace) noexcept=0
Register a plugin creator.
std::vector< int > mOutputDims
Definition: bert_infer.h:351
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: priorBoxPlugin.cpp:378
def test_backward(self)
Definition: clip_test.py:50
virtual IGatherLayer * addGather(ITensor &data, ITensor &indices, int32_t axis)=0
Add a gather layer to the network.
int mFeatureLength
Definition: multilevelCropAndResizePlugin.h:96
def __init__(self, graph, path=None, tensorboard_dir=None, engine_dir=None)
Definition: backend/tf/loader.py:308
@ kWHILE
Tensor is a scalar of type kBOOL. Loop terminates when value is false.
def test_raise(self)
Definition: quant_pooling_test.py:61
void operator()(void **ptr, size_t size)
Definition: sampleDevice.h:291
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: normalizePlugin.cpp:166
Weights deserializeToDevice(const char *&hostBuffer, size_t count)
Definition: gridAnchorPlugin.cpp:268
void clear() noexcept final
Clear the error stack on the error recorder.
Definition: ErrorRecorder.h:65
int mNbWeights
Definition: normalizePlugin.h:129
const char * getPluginVersion() const override
Return the plugin version.
Definition: multilevelProposeROIPlugin.cpp:216
static const char * severityPrefix(Severity severity)
returns an appropriate string for prefixing a log message with the given severity
Definition: pluginLogging.h:400
def compute_amax(self, str method, *int stride=1, int start_bin=128, float percentile=99.99)
Definition: histogram.py:214
void configurePlugin(const PluginTensorDesc *in, int nbInput, const PluginTensorDesc *out, int nbOutput) override
Definition: sampleUffPluginV2Ext.cpp:501
const char * getPluginVersion() const override
Return the plugin version.
Definition: nmsPlugin.cpp:154
bert::cuda_unique_ptr< void > mWordEmbDev
Definition: embLayerNormVarSeqlenPlugin.h:90
RNNGateType
Identifies an individual gate within an RNN cell.
Definition: NvInfer.h:3201
cudnnTensorDescriptor_t desc
Definition: groupNormalizationPlugin.h:103
const TKernelMeta * mKernelMeta
Definition: fused_multihead_attention.h:278
void initialize() override
it is called right before inference starts
Definition: textWriter.cpp:36
Weights mWeights
Definition: normalizePlugin.h:96
virtual float smallerThanMinimalLikelihood() const =0
std::set< int > mValidSequences
Definition: fused_multihead_attention.h:288
virtual IIteratorLayer * addIterator(ITensor &tensor, int32_t axis=0, bool reverse=false) noexcept=0
Return layer that subscripts tensor by loop iteration.
void setModelDtype(const nvinfer1::DataType mdt)
Set the Model Data Type.
Definition: sampleConfig.h:84
void serialize(void *buffer) const override
Serialize the layer.
Definition: generateDetectionPlugin.cpp:195
conv
Definition: demo/Tacotron2/waveglow/model.py:41
half scalbn(half arg, int exp)
Multiply by power of two.
Definition: ieee_half.h:2651
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: multilevelProposeROIPlugin.cpp:231
Definition: builder.py:88
def get_layer_class_mapping()
Definition: trt/util.py:41
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: multilevelCropAndResizePlugin.cpp:274
A Scale layer in a network definition.
Definition: NvInfer.h:2104
static std::vector< PluginField > mPluginAttributes
Definition: nmsPlugin.h:116
const std::string EMBED_NAME
Definition: sampleCharRNN.cpp:79
nvinfer1::ILayer * parseConvolution(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &weightFactory, BlobNameToTensor &tensors)
Definition: parseConv.cpp:23
virtual int line() const =0
source line at which the error occurred
static expr hypot(float x, float y)
Hypotenuse implementation.
Definition: samples/common/half.h:1863
hidden_size
Definition: modules/quant_rnn.py:388
n_mel_channels
Definition: demo/Tacotron2/tacotron2/model.py:591
once_logged
Definition: Polygraphy/polygraphy/logger/logger.py:129
def test_weight_fake_per_channel(self)
Definition: quant_linear_test.py:71
logger
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:28
def test_from_to_yaml(self)
Definition: tensor_quant_test.py:338
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: cropAndResizePlugin.cpp:235
void check_valid_inputs(const nvinfer1::Dims *inputs, int nbInputDims)
Definition: proposalLayerPlugin.cpp:236
static PluginFieldCollection mFC
Definition: normalizePlugin.h:125
hop_length
Definition: stft.py:47
BboxInfo(T conf_score, int label, int bbox_idx, bool kept)
Definition: bboxUtils.h:46
float minSize
Definition: NvInferPluginUtils.h:145
_model_bytes
Definition: backend/onnxrt/loader.py:30
static expr log2(float arg)
Binary logarithm implementation.
Definition: samples/common/half.h:1827
size_t nbBytes() const
Returns the size (in bytes) of the buffer.
Definition: buffers.h:136
std::vector< float > anchorsRatios
Definition: nvFasterRCNNPlugin.h:123
virtual nvinfer1::IPlugin * createPlugin(const char *layerName, const nvinfer1::Weights *weights, int32_t nbWeights)=0
Creates a plugin.
void configurePlugin(const nvinfer1::DynamicPluginTensorDesc *in, int nbInputs, const nvinfer1::DynamicPluginTensorDesc *out, int nbOutputs) override
Definition: instanceNormalizationPlugin.cpp:297
CUresult(* _cuLinkAddFile)(CUlinkState, CUjitInputType, const char *, unsigned int, CUjit_option *, void **)
Definition: cudaDriverWrapper.h:78
def __eq__(self, "Graph" other)
Definition: graph.py:151
void exportJSONOutput(const nvinfer1::IExecutionContext &context, const Bindings &bindings, const std::string &fileName)
Export output tensors to JSON file.
Definition: sampleReporting.cpp:335
const char * getPluginType() const override
Return the plugin type.
Definition: resizeNearestPlugin.cpp:165
const std::string RNNW_L0_NAME
Definition: sampleCharRNN.cpp:73
virtual void setBroadcastAcrossBatch(bool broadcastAcrossBatch)=0
Set whether to enable broadcast of tensor across the batch.
std::vector< std::string > mLayerNames
Definition: samples/common/common.h:221
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: priorBoxPlugin.cpp:185
nvinfer1::DimsHW sortClassScoreDims
Definition: maskRCNNKernels.h:167
int mNumInputs
Definition: flattenConcat.h:102
TrtCudaBuffer(size_t size)
Definition: sampleDevice.h:244
A
Definition: 07_creating_a_model_with_the_layer_api/generate.py:84
def _expand_number(m)
Definition: numbers.py:64
void addToModel(nvinfer1::INetworkDefinition *network, int maxInputSequenceLength, nvinfer1::ITensor *inputEmbeddedData, nvinfer1::ITensor *actualInputSequenceLengths, nvinfer1::ITensor **inputStates, nvinfer1::ITensor **memoryStates, nvinfer1::ITensor **lastTimestepStates) override
add the memory and last timestep states to the network lastTimestepHiddenStates is the pointer to the...
Definition: lstmEncoder.cpp:60
virtual ~IActivationLayer()
Definition: NvInfer.h:1663
basic_tokenizer
Definition: tokenization.py:166
static expr fdim(float x, float y)
Positive difference implementation.
Definition: samples/common/half.h:1734
def narrow_range(self)
Definition: tensor_quant.py:148
def test_quant_different_prec(self, verbose)
Definition: quant_rnn_test.py:223
const nvinfer1::PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: embLayerNormVarSeqlenPlugin.cpp:435
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: cropAndResizePlugin.cpp:241
def __init__(self, num_bits, axis, unsigned, num_bins=2048, grow_method=None, skip_zeros=False)
Definition: histogram.py:44
EntropyCalibratorImpl(TBatchStream stream, int firstBatch, std::string networkName, const char *inputBlobName, bool readCache=true)
Definition: EntropyCalibrator.h:31
nvinfer1::Dims mCHW
Definition: flattenConcat.h:103
@ kEQUAL
Check if two elements are equal.
detail::enable< half &, T >::type operator-=(T rhs)
Arithmetic assignment.
Definition: ieee_half.h:1133
def log(self, message, severity, mode=LogMode.EACH, stack_depth=2)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:155
expr log1p(half arg)
Natural logorithm.
Definition: ieee_half.h:2361
virtual std::string getInfo()=0
get the textual description of the component
virtual int getNbErrors() const =0
Get the number of errors that occurred during prior calls to parse.
virtual ~IGatherLayer()
Definition: NvInfer.h:2709
TrtCudaBuffer & operator=(const TrtCudaBuffer &)=delete
detail::enable< half &, T >::type operator/=(T rhs)
Arithmetic assignment.
Definition: ieee_half.h:1145
def test_equals(self)
Definition: test_ir.py:140
Bindings & mBindings
Definition: sampleInference.cpp:457
nvinfer1::Weights convertRNNBias(nvinfer1::Weights input)
Converts RNN Biases from TensorFlow's format to TensorRT's format.
Definition: sampleCharRNN.cpp:471
model
Definition: backend/trt_legacy.py:146
Weights copyToDevice(const void *hostData, size_t count)
Definition: priorBoxPlugin.cpp:214
virtual ITensor * addInput(const char *name, DataType type, Dims dimensions)=0
Add an input tensor to the network.
nvcaffeparser1::IPluginFactoryV2 * mPluginFactoryV2
Definition: caffeParser.h:88
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleUffSSD.cpp:97
TestAtom(bool started, const std::string &name, const std::string &cmdline)
Definition: pluginLogging.h:295
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleUffMNIST.cpp:56
int32_t preNmsTop
Definition: NvInferPluginUtils.h:121
def register_callback()
Definition: tools/Polygraphy/polygraphy/backend/onnx/__init__.py:5
Batch(ICudaEngine *engine, void *userInputPtr, void *itemInputPtr, const Args &args)
Definition: sampleMovieLensMPS.cpp:191
bool next() override
Definition: BatchStream.h:58
old_bytes
Definition: test_cuda.py:28
int initialize() override
Initialize the layer for execution.
static size_t serialized_size(std::vector< T > const &value)
Definition: serialize.hpp:91
std::string mNameSpace
Definition: specialSlicePlugin.h:96
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleMLP.cpp:255
int getInputDimensionSize() override
get the upper bound for the possible values of indices
Definition: slpEmbedder.cpp:59
bool mParamsOnDevice
Definition: skipLayerNormInt8InterleavedPlugin.h:81
def forward(self, forward_input)
Definition: demo/Tacotron2/waveglow/model.py:133
std::string mNamespace
Definition: instanceNormalizationPlugin.h:122
def amax(self)
Definition: tensor_quant.py:124
unsigned int fused_multihead_attention_v2_int8_256_64_kernel_cubin_len
Definition: fused_multihead_attention_v2_int8_256_64_kernel.sm72.cpp:20009
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: priorBoxPlugin.cpp:173
Definition: tf/test_loader.py:28
def load(self)
Definition: onnx_models.py:42
const void * readCalibrationCache(size_t &length) override
Definition: EntropyCalibrator.h:121
Definition: Polygraphy/polygraphy/logger/logger.py:43
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: resizeNearestPlugin.cpp:192
prenet
Definition: export_tacotron2_onnx.py:112
GridAnchorGenerator(const GridAnchorParameters *param, int numLayers, const char *version)
Definition: gridAnchorPlugin.cpp:36
Definition: quant_instancenorm.py:33
~LSTMDecoder() override=default
~FlattenConcatPluginCreator() override=default
float minBoxSize
Definition: NvInferPluginUtils.h:126
def compute_amax(self)
Definition: max.py:88
def parse_args(parser)
Definition: export_onnx2trt.py:28
def test_overflow_fp16(self)
Definition: tensor_quant_test.py:244
std::size_t mSize
Definition: samples/common/common.h:330
bool fallback
Definition: sampleOptions.h:138
float mScale
Definition: resizeNearestPlugin.h:115
path
Definition: backend/trt/loader.py:150
virtual void setPluginNamespace(const char *pluginNamespace)=0
Set the namespace that this plugin object belongs to.
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: reorgPlugin.cpp:51
_graph
Definition: backend/tf/loader.py:36
Definition: test_calibrator.py:38
int dims[4]
Definition: sampleReformatFreeIO.cpp:117
def test_against_unquantized(self)
Definition: quant_conv_transposed_test.py:497
void configure(const nvinfer1::Dims *inputDims, int nbInputs, const nvinfer1::Dims *outputDims, int nbOutputs, int maxBatchSize) override
Definition: debugUtil.cpp:44
static expr tanh(float arg)
Hyperbolic tangent implementation.
Definition: samples/common/half.h:1959
int enqueue(const nvinfer1::PluginTensorDesc *inputDesc, const nvinfer1::PluginTensorDesc *outputDesc, const void *const *inputs, void *const *outputs, void *workspace, cudaStream_t stream) override
Execute the layer.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:161
def forward(self, input)
Definition: quant_conv.py:204
_model
Definition: backend/onnx/loader.py:123
SampleUniquePtr< nvinfer1::ICudaEngine > mPreprocessorEngine
Definition: sampleDynamicReshape.cpp:84
def test_repr(self)
Definition: calibrator_test.py:294
int initialize() override
Initialize the layer for execution.
Definition: multilevelProposeROIPlugin.cpp:143
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: qkvToContextInt8InterleavedPlugin.cpp:211
std::vector< void * > mDeviceBindings
The vector of device buffers needed for engine execution.
Definition: buffers.h:458
def _compute_amax_mse(self, stride, start_bin)
Definition: histogram.py:166
def test_input_fake_quant_disable(self)
Definition: quant_pooling_test.py:96
static const int POOL_SIZE
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:29
def min(self, *args)
Definition: 08_replacing_a_subgraph/generate.py:23
config
Definition: backend/tf/loader.py:406
index
Definition: test_misc.py:38
void operator()(TrtCudaStream &stream) const
Definition: sampleInference.cpp:223
int seqSize
Definition: sampleCharRNN.cpp:118
std::string mPrefix
Definition: samples/common/logging.h:103
def free(self, ptr)
Definition: cuda.py:65
virtual ~IInt8EntropyCalibrator2()
Definition: NvInfer.h:6086
TextReader(std::shared_ptr< std::istream > textInput, Vocabulary::ptr vocabulary)
Definition: textReader.cpp:26
half nanh(const char *)
Get NaN value.
Definition: ieee_half.h:2316
def set_tensor_name(tensor, prefix, name)
Definition: builder.py:106
int getTailWithNoWorkRemaining()
Definition: beamSearchPolicy.cpp:119
std::string weightsFile
The filename of the weights file.
Definition: sampleMLP.cpp:50
static half fabs(half arg)
Absolute value implementation.
Definition: samples/common/half.h:2524
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleMNISTAPI.cpp:95
std::vector< size_t > mInputSizes
Definition: bert_infer.h:349
void setCudaDevice(int device, std::ostream &os)
Definition: sampleDevice.h:354
virtual void addVerbosity()=0
Increase the Verbosity Level.
@ kEXPLICIT_PRECISION
Setting the network to be an explicit precision network has the following implications: 1) Precision ...
def validate_case_matches_checkpoint(do_lower_case, init_checkpoint)
Definition: tokenization.py:27
path
Definition: backend/onnx/loader.py:32
nvinfer1::DimsExprs getOutputDimensions(int outputIndex, const nvinfer1::DimsExprs *inputs, int nbInputs, nvinfer1::IExprBuilder &exprBuilder) override
Definition: skipLayerNormInt8InterleavedPlugin.cpp:110
sample::Logger::TestResult build()
Builds the network engine.
Definition: sampleINT8API.cpp:490
std::string getInfo() override
get the textual description of the component
Definition: slpEmbedder.cpp:64
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: specialSlicePlugin.cpp:145
def test_raise(self)
Definition: quant_pooling_test.py:109
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleMovieLens.cpp:151
virtual void addVerbosity()
Increase the Verbosity Level.
Definition: parserOnnxConfig.h:99
nvinfer1::ITensor * cell
Definition: sampleCharRNN.cpp:245
const char * getPluginName() const override
Return the plugin name.
Definition: generateDetectionPlugin.cpp:51
def __call__(self, *args, **kwargs)
Definition: backend/trt/loader.py:52
int H
Definition: nvFasterRCNNPlugin.h:99
static const float DETECTION_MIN_CONFIDENCE
Definition: tlt_mrcnn_config.h:44
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: regionPlugin.cpp:466
def __init__(self, waveglow, filter_length=1024, n_overlap=4, win_length=1024, mode='zeros')
Definition: denoiser.py:26
static const char * EMB_LAYER_NORM_VAR_SEQLEN_NAME
Definition: embLayerNormVarSeqlenPlugin.cpp:48
def __enter__(self)
Definition: Tacotron2/inference.py:170
def test_can_copy_outputs_from_other_node(self)
Definition: test_ir.py:74
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: priorBoxPlugin.cpp:340
Definition: backend/trt/loader.py:235
virtual ILRNLayer * addLRN(ITensor &input, int32_t window, float alpha, float beta, float k)=0
Add a LRN layer to the network.
max_batch_size
Definition: backend/tf/loader.py:229
Int8EntropyCalibrator2(TBatchStream stream, int firstBatch, const char *networkName, const char *inputBlobName, bool readCache=true)
Definition: EntropyCalibrator.h:105
def infer_impl(self, feed_dict)
Definition: backend/onnxrt/runner.py:47
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:60
def _check_is_max_context(doc_spans, cur_span_index, position)
Definition: data_processing.py:49
virtual void write(const int *hOutputData, int actualOutputSequenceLength, int actualInputSequenceLength)=0
write the generated sequence
bool constructNetwork(SampleUniquePtr< nvinfer1::IBuilder > &builder, SampleUniquePtr< nvinfer1::INetworkDefinition > &network, SampleUniquePtr< nvinfer1::IBuilderConfig > &config, SampleUniquePtr< nvuffparser::IUffParser > &parser)
Parses an UFF model for SSD and creates a TensorRT network.
Definition: sampleUffSSD.cpp:176
decoder_rnn
Definition: demo/Tacotron2/tacotron2/model.py:267
def spectral_normalize(self, magnitudes)
Definition: layers.py:70
def empty()
Definition: tensor.py:151
Definition: gridAnchorPlugin.h:102
virtual IConstantLayer * addConstant(Dims dimensions, Weights weights)=0
Add a constant layer to the network.
bool buildPreprocessorEngine(const SampleUniquePtr< nvinfer1::IBuilder > &builder)
Builds an engine for preprocessing (mPreprocessorEngine).
Definition: sampleDynamicReshape.cpp:127
std::vector< void * > mDeviceBuffers
Definition: bert_infer.h:347
def register(opsets=None)
Definition: graph.py:69
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: nvFasterRCNNPlugin.cpp:262
int getSMVersion() const
Definition: qkvToContextInt8InterleavedPlugin.cpp:83
float enqEnd
Definition: sampleReporting.h:93
def forward(self, z)
Definition: demo/Tacotron2/waveglow/model.py:53
int pW
Definition: sampleUffPluginV2Ext.cpp:183
b
Definition: 03_isolating_a_subgraph/generate.py:31
const char * getPluginVersion() const override
Return the plugin version.
Definition: proposalLayerPlugin.cpp:170
~MultilevelProposeROI() override=default
const char * NMS_PLUGIN_NAMES[]
Definition: batchedNMSPlugin.cpp:35
def quant_weight_inplace(model)
Definition: helper.py:112
std::unordered_map< std::string, int > getBindings(bool(*predicate)(const Binding &b)) const
Definition: sampleUtils.h:436
bool supportsFormatCombination(int pos, const PluginTensorDesc *inOut, int nbInputs, int nbOutputs) const override
The combination of kLINEAR + kINT8/kHALF/kFLOAT is supported.
Definition: sampleUffPluginV2Ext.cpp:521
def test_initialize_deactivate(self)
Definition: quant_modules_test.py:73
unsigned char fused_multihead_attention_v2_int8_256_64_kernel_sm80_cubin[]
Definition: fused_multihead_attention_v2_int8_256_64_kernel.sm80.cpp:19
intermediate_size
Definition: builder_varseqlen.py:95
const char * RESIZE_PLUGIN_VERSION
Definition: resizeNearestPlugin.cpp:32
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleUffFasterRCNN.cpp:316
virtual ITopKLayer * addTopK(ITensor &input, TopKOperation op, int32_t k, uint32_t reduceAxes)=0
Add a TopK layer to the network.
virtual IPluginV2Layer * addPluginV2(ITensor *const *inputs, int32_t nbInputs, IPluginV2 &plugin)=0
Add a plugin layer to the network using the IPluginV2 interface.
Functor to enqueue inference with implict batch.
Definition: sampleInference.cpp:191
int32_t poolingH
Definition: NvInferPluginUtils.h:118
virtual ~ISoftMaxLayer()
Definition: NvInfer.h:2196
std::ostream & mOutput
Definition: pluginLogging.h:98
def get_mel_audio_pair(self, filename)
Definition: waveglow/data_function.py:41
Definition: comparator/struct.py:49
Graph import_graph(onnx.GraphProto onnx_graph, "OrderedDict[str, Tensor]" tensor_map=None, opset=None)
Definition: onnx_importer.py:155
Definition: sampleUtils.h:304
head_size
Definition: builder_varseqlen.py:97
std::string inputNodeName
Definition: sampleUffFasterRCNN.cpp:62
timestamp
Definition: Polygraphy/polygraphy/logger/logger.py:132
def test_against_unquantized(self)
Definition: quant_conv_test.py:210
UnaryOperation
Enumerates the unary operations that may be performed by a Unary layer.
Definition: NvInfer.h:3478
location_dense
Definition: demo/Tacotron2/tacotron2/model.py:37
nvinfer1::DimsHW argMaxLabelDims
Definition: maskRCNNKernels.h:99
InferenceTrace getTrace(const TimePoint &cpuStart, const TrtCudaEvent &gpuStart, bool skipTransfers)
Definition: sampleInference.cpp:418
matmul_v_input_quantizer
Definition: quant_bert.py:75
The ManagedBuffer class groups together a pair of corresponding device and host buffers.
Definition: buffers.h:222
preprocessor
Definition: backend/trt_legacy.py:52
virtual const char * getFullTextFileName() const =0
Get the File Name of the Network Description as a Text File, including the weights.
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: coordConvACPlugin.cpp:190
def __call__(self)
Definition: backend/onnx/loader.py:76
bool refit
Definition: sampleOptions.h:174
std::vector< float > mResizedKernelWeights
Definition: slpProjection.h:55
def __call__(self)
Definition: backend/tf/loader.py:179
def __call__(self)
Definition: backend/tf/loader.py:66
const char * mPluginNamespace
Definition: groupNormalizationPlugin.h:95
void setReportableSeverity(Severity severity)
Method for controlling the verbosity of logging output.
Definition: pluginLogging.h:275
int32_t useDLACore
Definition: argsParser.h:84
virtual const char * getPluginType() const =0
Return the plugin type.
def test_state_loading(self)
Definition: tensor_quantizer_test.py:139
int32_t keepTopK
Definition: NvInferPluginUtils.h:250
def test_cleanup_remove_unused_node_outputs(self)
Definition: test_ir.py:638
def __getitem__(self, index)
Definition: tacotron2/data_function.py:76
@ kDISABLE_TIMING_CACHE
Disable reuse of timing information across identical layers.
Dims mDims
Definition: BatchStream.h:149
float mEps
Definition: normalizePlugin.h:128
input_metadata
Definition: comparator/data_loader.py:91
The TensorRT Caffe parser API namespace.
bool processInput(const samplesCommon::BufferManager &buffers)
Reads the input and mean data, preprocesses, and stores the result in a managed buffer.
Definition: sampleUffFasterRCNN.cpp:378
void readPPMFile(const std::string &filename, samplesCommon::PPM< C, H, W > &ppm)
Definition: samples/common/common.h:673
float x2
Definition: sampleUffMaskRCNN.cpp:54
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: batchedNMSPlugin.cpp:266
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: cropAndResizePlugin.cpp:113
def __str__(self)
Definition: backend/trt/loader.py:250
Selects Algorithms with minimum workspace requirements.
Definition: sampleAlgorithmSelector.cpp:275
def forward(self, inputs)
Definition: demo/Tacotron2/tacotron2/model.py:648
@ kLAST_VALUE
Output value is value of tensor for last iteration.
std::array< TrtCudaStream, static_cast< int >(StreamType::kNUM)> MultiStream
Definition: sampleInference.cpp:271
Definition: test_postprocess.py:20
def infer_impl(self)
Definition: backend/base/runner.py:96
~SpecialSlice() override=default
nvinfer1::ILayer * parseInnerProduct(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &weightFactory, BlobNameToTensor &tensors)
Definition: parseInnerProduct.cpp:23
std::string loadEngine
Definition: sampleCharRNN.cpp:125
exclude_outputs
Definition: backend/trt/loader.py:205
nvinfer1::Dims mImageSize
Definition: multilevelProposeROIPlugin.h:113
TRTParsedArgs is returned by getOptions after it has parsed a command line argument list (argv).
Definition: getOptions.h:57
@ kNONE
Treat x as a matrix if it has two dimensions, or as a collection of matrices if x has more than two d...
void check_valid_inputs(const nvinfer1::Dims *inputs, int nbInputDims)
Definition: pyramidROIAlignPlugin.cpp:144
static expr cosh(float arg)
Hyperbolic cosine implementation.
Definition: samples/common/half.h:1951
std::string mImageFilename
Definition: sampleConfig.h:40
fc2
Definition: tools/pytorch-quantization/tests/fixtures/models.py:50
Definition: backend/tf/runner.py:30
def identity_builder_network()
Definition: test_calibrator.py:32
~MultiplicativeAlignment() override=default
nvinfer1::Weights mBiasWeights
Definition: samples/opensource/samplePlugin/fcPlugin.h:309
int mNumLayers
Definition: lstmDecoder.h:50
data_loader
Definition: comparator/data_loader.py:200
float sync(const TimePoint &cpuStart, const TrtCudaEvent &gpuStart, std::vector< InferenceTrace > &trace, bool skipTransfers)
Definition: sampleInference.cpp:336
_serialized_engine
Definition: backend/trt/loader.py:469
float mFGThreshold
Definition: multilevelProposeROIPlugin.h:98
def forward(self, inputs)
Definition: tensor_quantizer.py:317
bias
Definition: modules/quant_rnn.py:389
def main()
Definition: builder_varseqlen.py:581
unsigned char fused_multihead_attention_fp16_96_64_kernel_sm75_cu_o[]
Definition: fused_multihead_attention_fp16_96_64_kernel.sm75.cpp:19
def _is_punctuation(char)
Definition: tokenization.py:416
nvinfer1::IExecutionContext & mContext
Definition: sampleInference.cpp:183
void dumpInputs(const nvinfer1::IExecutionContext &context, std::ostream &os) const
Definition: sampleUtils.h:383
@ kVECTOR
Treat x as a vector if it has one dimension, or as a collection of vectors if x has more than one dim...
static std::basic_ostream< charT, traits > & write(std::basic_ostream< charT, traits > &out, float arg)
Output implementation.
Definition: ieee_half.h:1245
The Anchor Generator plugin layer generates the prior boxes of designated sizes and aspect ratios acr...
Definition: NvInferPluginUtils.h:143
int scoresSize
Definition: batchedNMSPlugin.h:114
std::string mPluginNamespace
Definition: caffeParser.h:92
def test_against_unquantized(self)
Definition: quant_conv_test.py:532
Definition: backend/tf/loader.py:138
std::vector< std::vector< float > > mTimes
Definition: bert_infer.h:353
def __init__(self, model, input_metadata, output_names, name=None)
Definition: backend/pyt/runner.py:22
def get_quantile(self)
Definition: demo/BERT/helpers/calibrator.py:101
const char * getPluginName() const override
Return the plugin name.
Definition: flattenConcat.cpp:301
void log(Severity severity, const char *msg) override
Implementation of the nvinfer1::ILogger::log() virtual method.
Definition: samples/common/logging.h:244
save_inputs_path
Definition: comparator/data_loader.py:202
int getBatchesRead() const
Definition: frcnnUtils.h:309
CUresult cuModuleLoadData(CUmodule *module, const void *image) const
Definition: cudaDriverWrapper.cpp:96
bool help
Definition: demo/BERT/infer_c/common.h:28
nvinfer1::DataType getDataType() const
Definition: caffeWeightFactory.cpp:68
bool build()
Builds the network engine.
Definition: sampleMovieLens.cpp:164
std::string referenceFileName
Definition: sampleINT8API.cpp:66
def test_can_copy_inputs_from_other_node(self)
Definition: test_ir.py:177
@ kBIAS
bias for IConvolutionLayer, IDeconvolutionLayer, or IFullyConnectedLayer
virtual void setVerbosityLevel(Verbosity v)
Definition: sampleConfig.h:125
virtual int64_t getImplementation() const =0
Return implementation of the algorithm.
nvinfer1::Dims mInputDims
The dimensions of the input to the network.
Definition: sampleUffSSD.cpp:93
void fill()
Definition: sampleUtils.h:232
void parse(Arguments &arguments) override
Definition: sampleOptions.cpp:734
def test_tensors_does_not_modify_tensors(self)
Definition: test_ir.py:563
std::string helpText
Text to show when printing out the command usage.
Definition: getOptions.h:39
void initialize() override
it is called right before inference starts
Definition: bleuScoreWriter.cpp:183
Definition: sampleCharRNN.cpp:71
Definition: embLayerNormVarSeqlenPlugin.h:43
_engine
Definition: backend/trt/loader.py:501
def forward(self, input, hx=None)
Definition: modules/quant_rnn.py:185
def test_extend_1_args_1_rv(self)
Definition: test_func.py:80
constexpr int32_t EnumMax< LayerType >()
Maximum number of elements in LayerType enum.
Definition: NvInfer.h:484
static PluginFieldCollection mFC
Definition: multilevelProposeROIPlugin.h:138
def __init__(self, shape, dtype=trt.int32)
Definition: infer.py:33
def f1_score(prediction, ground_truth)
Definition: evaluate-v1.1.py:47
void runInference(const InferenceOptions &inference, InferenceEnvironment &iEnv, int device, std::vector< InferenceTrace > &trace)
Run inference and collect timing.
Definition: sampleInference.cpp:557
std::string mNamespace
Definition: skipLayerNormInt8InterleavedPlugin.h:116
unsigned char fused_multihead_attention_v2_int8_128_64_kernel_cubin[]
Definition: fused_multihead_attention_v2_int8_128_64_kernel.sm72.cpp:19
def infer_impl(self, feed_dict)
Definition: backend/cntk/runner.py:39
size_t mCropHeight
Definition: cropAndResizePlugin.h:101
def step_size(self)
Definition: tensor_quantizer.py:132
int h
Definition: samples/common/common.h:655
def test_save_timeline(self)
Definition: tf/test_runner.py:42
severity
Definition: Polygraphy/polygraphy/logger/logger.py:62
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: reorgPlugin.cpp:148
BLEUScoreWriter(std::shared_ptr< std::istream > referenceTextInput, Vocabulary::ptr vocabulary, int maxOrder=4)
Definition: bleuScoreWriter.cpp:150
imported_graph
Definition: test_api.py:27
Definition: fused_multihead_attention.h:282
virtual IRecurrenceLayer * addRecurrence(ITensor &initialValue) noexcept=0
Create a recurrence layer for this loop with initialValue as its first input.
bool teardown()
Used to clean up any state created in the sample class.
Definition: sampleUffMNIST.cpp:321
@ kWRAP
Coordinates wrap around periodically.
root_dir
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:102
float getMaxValue(const float *buffer, int64_t size)
Definition: samples/common/common.h:501
@ kUNSUPPORTED_STATE
An error occurred due to the network not being supported on the device due to constraints of the hard...
n_mel_channels
Definition: layers.py:60
def export_graph(Graph graph)
Definition: base_exporter.py:21
Layer that represents a parametric ReLU operation.
Definition: NvInfer.h:4320
def test_against_unquantized(self, verbose)
Definition: quant_rnn_test.py:133
void ** getDeviceBuffers()
Definition: sampleUtils.h:334
def convert_rnn_bias(weights, dimensions, forget_bias=1.0)
Definition: chptToBin.py:285
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: priorBoxPlugin.cpp:302
@ kHWC
Non-vectorized channel-last format.
cpu_run
Definition: Tacotron2/inference.py:168
cudnnTensorDescriptor_t mSrcDescriptor
Definition: sampleUffPluginV2Ext.cpp:632
def __init__(self, explicit_precision=None, explicit_batch=None)
Definition: backend/trt/loader.py:61
def __init__(self, input_size, hidden_size, bias=True)
Definition: modules/quant_rnn.py:426
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: regionPlugin.cpp:403
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: batchedNMSPlugin.cpp:331
def test_broadcast(self)
Definition: functional_test.py:59
int32_t getTensorRTVersion() const
Return the API version with which this plugin was built.
Definition: NvInferRuntimeCommon.h:692
enable< bool, T, U >::type operator>(T x, U y)
Comparison for greater than.
Definition: ieee_half.h:2148
TrtDeviceBuffer mDeviceBuffer
Definition: sampleDevice.h:351
ProposalPluginCreator()
Definition: proposalPlugin.cpp:317
int vocabSize
Definition: sampleCharRNN.cpp:120
std::unordered_map< std::string, int > getOutputBindings() const
Definition: sampleUtils.h:424
virtual DataType getBindingDataType(int32_t bindingIndex) const noexcept=0
Determine the required data type for a buffer from its binding index.
def convert_rnn_kernel(weights, dimensions, is_decoder_rnn=False)
Definition: chptToBin.py:199
Enqueue(nvinfer1::IExecutionContext &context, void **buffers)
Definition: sampleInference.cpp:177
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: multilevelCropAndResizePlugin.cpp:111
def test_get_used_node_ids(self, graph)
Definition: test_ir.py:600
const char * PRIOR_BOX_PLUGIN_VERSION
Definition: priorBoxPlugin.cpp:32
def get_input_metadata(network)
Definition: trt/util.py:80
Definition: export_tacotron2_onnx.py:103
bool readReferenceFile(const std::string &fileName, std::vector< std::string > &refVector)
Definition: samples/common/common.h:408
virtual int getSourceStatesSize()=0
get the size of the source states
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleUffFasterRCNN.cpp:96
def GRUCell(input, hidden, w_ih, w_hh, b_ih=None, b_hh=None)
Definition: _functions/quant_rnn.py:74
Definition: sampleUffMaskRCNN.cpp:52
bool mTF32
Definition: sampleConfig.h:52
static const char * QKV_TO_CONTEXT_INTERLEAVED_PLUGIN_VERSION
Definition: qkvToContextInt8InterleavedPlugin.cpp:39
expr fma(half x, half y, half z)
Fused multiply add.
Definition: ieee_half.h:2275
int calBatchSize
Definition: sampleUffFasterRCNN.cpp:79
measurements
Definition: Tacotron2/inference.py:166
virtual void setErrorRecorder(nvinfer1::IErrorRecorder *recorder)=0
Set the ErrorRecorder for this interface.
def forward(self, attention_weights_cat)
Definition: demo/Tacotron2/tacotron2/model.py:41
def __len__(self)
Definition: cmudict.py:47
def htod(self, dst, src, nbytes, stream=None)
Definition: cuda.py:69
def AutogradRNN(mode, input_size, hidden_size, num_layers=1, batch_first=False, dropout=0, train=True, bidirectional=False, variable_length=False, dropout_state=None, flat_weight=None, input_quantizers=None, weight_quantizers=None)
Definition: _functions/quant_rnn.py:236
Definition: postprocess.py:20
static const int RPN_ANCHOR_STRIDE
Definition: tlt_mrcnn_config.h:73
a
Definition: 03_isolating_a_subgraph/generate.py:30
void inferenceLoop(IterationStreams &iStreams, const TimePoint &cpuStart, const TrtCudaEvent &gpuStart, int iterations, float maxDurationMs, float warmupMs, std::vector< InferenceTrace > &trace, bool skipTransfers)
Definition: sampleInference.cpp:476
nvinfer1::Weights mKernelWeights
Definition: multiplicativeAlignment.h:52
def __init__(self, tacotron2)
Definition: export_tacotron2_onnx.py:77
const char * getPluginName() const
Return the plugin name.
Definition: split.h:142
nvinfer1::DataType mModelDtype
Definition: sampleConfig.h:51
bool runInFp16
Definition: argsParser.h:82
void reportAlgorithms(const nvinfer1::IAlgorithmContext *const *algoContexts, const nvinfer1::IAlgorithm *const *algoChoices, int32_t nbAlgorithms) override
Called by TensorRT to report choices it made.
Definition: sampleAlgorithmSelector.cpp:168
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleINT8.cpp:98
def __call__(self, builder, network)
Definition: backend/trt/loader.py:334
Definition: debugUtil.h:38
static void deserialize(void const **buffer, size_t *buffer_size, T *value)
Definition: serialize.hpp:56
end
Definition: demo/Tacotron2/waveglow/model.py:108
~SpecialSlicePluginCreator() override=default
virtual void setInput(int32_t index, ITensor &tensor)=0
Replace an input of this layer with a specific tensor.
Definition: quant_conv_transposed_test.py:48
def fake_quant(self)
Definition: tensor_quantizer.py:143
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: pyramidROIAlignPlugin.cpp:201
static const float DETECTION_NMS_THRESHOLD
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:47
virtual const char * getTextFileName() const =0
Returns the File Name of the Network Description as a Text File.
virtual void setModelDtype(const nvinfer1::DataType)=0
Set the Model Data Type.
float scoreThreshold
Definition: maskRCNNKernels.h:87
SoftmaxLikelihoodCombinationOperator()=default
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: detectionLayerPlugin.cpp:57
half nearbyint(half arg)
Nearest integer using half's internal rounding mode.
Definition: ieee_half.h:2586
y
Definition: 06_removing_nodes/generate.py:30
virtual const char * getPluginVersion() const =0
Return the plugin version.
Definition: batchTilePlugin.h:84
virtual TensorLocation getLocation() const =0
Get the storage location of a tensor.
nvinfer1::ILayer * parseClip(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &, BlobNameToTensor &tensors)
Definition: parseClip.cpp:23
void setPluginNamespace(const char *libNamespace)
Set the namespace of the plugin creator based on the plugin library it belongs to.
Definition: split.h:169
is_dynamic_op
Definition: backend/tf/loader.py:230
int32_t h() const
Get the height.
Definition: NvInfer.h:134
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleMNIST.cpp:291
Definition: onnx-graphsurgeon/tests/test_examples.py:33
static std::vector< PluginField > mPluginAttributes
Definition: cropAndResizePlugin.h:124
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: proposalPlugin.cpp:165
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: lReluPlugin.cpp:51
void printEpilog(std::vector< InferenceTime > timings, float walltimeMs, float percentile, int queries, std::ostream &os)
Definition: sampleReporting.cpp:112
def ultra_verbose(self, message, mode=LogMode.EACH)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:219
std::string mNamespace
Definition: batchTilePlugin.h:81
static bool isgreater(half x, half y)
Comparison implementation.
Definition: samples/common/half.h:2408
NetworkDefinitionCreationFlag
List of immutable network properties expressed at network creation time. NetworkDefinitionCreationFla...
Definition: NvInfer.h:6891
Definition: test_ir.py:121
int32_t inputOrder[3]
Definition: NvInferPluginUtils.h:187
def audio2mel(dataset_path, audiopaths_and_text, melpaths_and_text, args)
Definition: preprocess_audio2mel.py:56
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: multilevelCropAndResizePlugin.cpp:300
def run(runners, data_loader=None, warm_up=None, use_subprocess=None, subprocess_timeout=None, subprocess_polling_interval=None, save_inputs_path=None)
Definition: comparator/comparator.py:33
DumpTensorPlugin(std::shared_ptr< std::ostream > out)
Definition: debugUtil.cpp:28
bool islessequal(half x, half y)
Comparison for less equal.
Definition: ieee_half.h:2806
void copyDeviceInputToFP32(const void *src, void *&dst)
Definition: sampleUffPluginV2Ext.cpp:594
def __init__(self)
Definition: cuda.py:91
static std::vector< PluginField > mPluginAttributes
Definition: multilevelCropAndResizePlugin.h:125
~Context() override=default
def forward(ctx, inputs, amax, num_bits=8, unsigned=False, narrow_range=True)
Definition: tensor_quant.py:248
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: specialSlicePlugin.cpp:57
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: specialSlicePlugin.cpp:194
std::unordered_map< uint64_t, const std::unique_ptr< TFusedMHAKernelList > > mKernels
Definition: fused_multihead_attention.h:327
def try_get_stream_handle(stream)
Definition: cuda.py:107
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: batchedNMSPlugin.cpp:197
name
Definition: test_comparator.py:102
nvinfer1::DimsHW sortNMSMarkDims
Definition: maskRCNNKernels.h:172
def __str__(self)
Definition: tensor_quant.py:152
virtual void setPluginNamespace(const char *libNamespace)=0
Set the namespace used to lookup and create plugins in the network.
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: resizeNearestPlugin.cpp:131
expr log(half arg)
Natural logorithm.
Definition: ieee_half.h:2347
int mInputWidth
Definition: multilevelCropAndResizePlugin.h:100
virtual ISoftMaxLayer * addSoftMax(ITensor &input)=0
Add a SoftMax layer to the network.
def check_model(model)
Definition: onnx/util.py:24
The Permute plugin layer permutes the input tensor by changing the memory order of the data.
Definition: NvInferPluginUtils.h:66
expr lgamma(half arg)
Natural logarithm of gamma function.
Definition: ieee_half.h:2533
_graph
Definition: backend/onnx/loader.py:67
unsigned int fused_multihead_attention_v2_fp16_256_64_kernel_sm80_cubin_len
Definition: fused_multihead_attention_v2_fp16_256_64_kernel.sm80.cpp:5427
DataReader::ptr gOriginalDataReader
Definition: limitedSamplesDataReader.h:42
Definition: export_tacotron2_onnx.py:76
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: resizeNearestPlugin.cpp:252
virtual ~ICudaEngine()
Definition: NvInferRuntime.h:1325
Definition: maskRCNNKernels.h:119
def main()
Definition: perf_varseqlen.py:39
Definition: Polygraphy/polygraphy/util/misc.py:432
LogStreamConsumer gLogError
Definition: logger.cpp:26
def parse_args(parser)
Definition: test_infer.py:36
fake_tensor_quant
Definition: tensor_quant.py:427
int32_t selectAlgorithms(const nvinfer1::IAlgorithmContext &context, const nvinfer1::IAlgorithm *const *choices, int32_t nbChoices, int32_t *selection) override
Return value in [0, nbChoices] for a valid algorithm.
Definition: sampleAlgorithmSelector.cpp:61
virtual void reduceVerbosity()=0
Decrease verbosity Level.
@ kNEAREST
ND (0 < N <= 8) nearest neighbor resizing.
def __exit__(self, exc_type, exc_value, traceback)
Definition: graph.py:54
static std::vector< PluginField > mPluginAttributes
Definition: proposalPlugin.h:144
SampleMovieLens(const SampleMovieLensParams ¶ms)
Definition: sampleMovieLens.cpp:89
@ kINVALID_CONFIG
An error occurred when comparing the state of an argument relative to other arguments.
def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros', **kwargs)
Definition: quant_conv.py:95
nvinfer1::ILayer * parseTanH(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &, BlobNameToTensor &tensors)
Definition: parseTanH.cpp:23
nvinfer1::Dims mImageSize
Definition: generateDetectionPlugin.h:107
int32_t numMinSize
Definition: NvInferPluginUtils.h:94
all_head_size
Definition: quant_bert.py:61
bool fp16
Allow running the network in FP16 mode.
Definition: argsParser.h:41
bool validateOutput(int digit)
Checks whether the model prediction (in mOutput) is correct.
Definition: sampleDynamicReshape.cpp:405
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: sampleUffPluginV2Ext.cpp:424
__host__ __forceinline__ reduced_divisor(int _y)
Definition: reducedMathPlugin.h:58
_graph
Definition: backend/tf/loader.py:225
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: multilevelProposeROIPlugin.cpp:466
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: multilevelCropAndResizePlugin.cpp:101
@ kLINSPACE
Generate evenly spaced numbers over a specified interval.
output_names
Definition: backend/pyt/runner.py:40
void serializeFromDevice(char *&hostBuffer, Weights deviceWeights) const
def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0, output_padding=0, groups=1, bias=True, dilation=1, padding_mode='zeros', **kwargs)
Definition: quant_conv.py:355
bool verifyOutput(const samplesCommon::BufferManager &buffers)
Filters output detections, handles post-processing of bounding boxes and verify results.
Definition: sampleFasterRCNN.cpp:303
const char * getPluginVersion() const override
Return the plugin version.
Definition: cropAndResizePlugin.cpp:258
def _apply(self, fn)
Definition: modules/quant_rnn.py:142
size_t getWorkspaceSize(const PluginTensorDesc *inputs, int nbInputs, const PluginTensorDesc *outputs, int nbOutputs) const override
Definition: batchedNMSPlugin.cpp:203
void parse(Arguments &arguments) override
Definition: sampleOptions.cpp:662
Definition: pluginLogging.h:32
Layer that represents an unary operation.
Definition: NvInfer.h:3517
virtual ~IParser()
Definition: NvOnnxParser.h:226
const std::vector< void * > & getDeviceBindings() const
Returns a vector of device buffers.
Definition: buffers.h:286
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: split.h:58
SampleMLP(const SampleMLPParams ¶ms)
Definition: sampleMLP.cpp:63
void memcpyBuffers(const bool copyInput, const bool deviceToHost, const bool async, const cudaStream_t &stream=0)
Definition: buffers.h:435
void setInputData()
Definition: sampleInference.cpp:371
static const int MIN_LEVEL
Definition: tlt_mrcnn_config.h:60
std::string getInfo() override
get the textual description of the component
Definition: benchmarkWriter.cpp:54
Definition: detectionLayerPlugin.h:35
def test_fake_quant_per_tensor(self)
Definition: quant_conv_transposed_test.py:126
Helper for creating a recurrent subgraph.
Definition: NvInfer.h:4645
@ kSUCCESS
Execution completed successfully.
@ kCOSH
Hyperbolic cosine.
bool checkBlobs(const trtcaffe::LayerParameter &msg, int bottoms, int tops)
Definition: opParsers.h:32
def deactivate_impl(self)
Definition: backend/tf/runner.py:73
std::string prototxtFileName
Filename of prototxt design file of a network.
Definition: argsParser.h:53
virtual ~PinnedHostBuffer()
Definition: pinnedHostBuffer.h:43
def test_input_fake_quant(self)
Definition: quant_pooling_test.py:159
bool output
Definition: sampleOptions.h:175
bool mDebugBuilder
Definition: sampleConfig.h:55
def test_opset(self)
Definition: onnx/test_loader.py:47
def __init__(self, outputs=None, runtime=None, runner_name=None)
Definition: comparator/struct.py:23
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: samplePlugin.cpp:58
def test_calibrator_with_path_name_cache(self, identity_builder_network)
Definition: test_calibrator.py:90
virtual const char * desc() const =0
description of the error
static std::vector< nvinfer1::PluginField > mPluginAttributes
Definition: qkvToContextInt8InterleavedPlugin.h:124
constexpr int defaultAvgRuns
Definition: sampleOptions.h:52
def test_no_quant(self)
Definition: quant_instancenorm_test.py:151
PaddingMode
Enumerates the modes of padding to perform in convolution, deconvolution and pooling layer,...
Definition: NvInfer.h:1170
def transpose_for_scores(self, x)
Definition: quant_bert.py:78
BertInferenceRunner(const std::string &enginePath, const int maxBatchSize, const int maxSeqLength, const bool enableGraph)
Definition: infer_c.cpp:25
Definition: quant_pooling.py:1
_graph
Definition: backend/tf/loader.py:322
static const struct bert::FusedMultiHeadAttentionKernelMetaInfoV2 sMhaKernelMetaInfosV2[]
int getBlobsSize(const std::string &layerName)
Definition: caffeWeightFactory.cpp:93
once_logged
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:103
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: proposalPlugin.cpp:276
def test_io_cannot_be_sync_list_on_init(self)
Definition: test_ir.py:776
def __init__(self)
Definition: comparator/struct.py:50
UffInput uffInputs
Definition: sampleOptions.h:99
size_t totalSize
Definition: maskRCNNKernels.h:116
def avgData(data, avg, times)
Definition: tracer.py:74
float _epsilon
Definition: instanceNormalizationPlugin.h:90
The BufferManager class handles host and device buffer allocation and deallocation.
Definition: buffers.h:237
int warmUpRuns
Definition: demo/BERT/infer_c/common.h:33
def test_per_tensor_scale(self)
Definition: tensor_quantizer_test.py:58
bool transposeSubBuffers(void *input, DataType type, int32_t num, int32_t height, int32_t width)
Transpose num sub-buffers of height * width.
unsigned char fused_multihead_attention_int8_128_64_kernel_sm75_cu_o[]
Definition: fused_multihead_attention_int8_128_64_kernel.sm75.cpp:20
_severity
Definition: Polygraphy/polygraphy/logger/logger.py:126
clip
Definition: functional.py:63
void setPluginFactoryV2(nvcaffeparser1::IPluginFactoryV2 *factory) override
Set the IPluginFactoryV2 used to create the user defined pluginV2 objects.
Definition: caffeParser.h:55
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: nvFasterRCNNPlugin.cpp:304
Allows a serialized functionally unsafe engine to be deserialized.
Definition: NvInferRuntime.h:726
constexpr int defaultWorkspace
Definition: sampleOptions.h:36
static const float DETECTION_NMS_THRESHOLD
Definition: tlt_mrcnn_config.h:47
~DumpTensorPlugin() override=default
def copy_to(self, host_buffer, stream=None)
Definition: cuda.py:171
bool build()
Definition: sampleUffMaskRCNN.cpp:352
def forward(self, x)
Definition: demo/Tacotron2/tacotron2/model.py:116
do_shape_inference
Definition: backend/onnx/loader.py:124
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: coordConvACPlugin.cpp:144
std::vector< std::pair< int32_t, int32_t > > formats
Definition: sampleAlgorithmSelector.cpp:241
def i(self, tensor_idx=0, producer_idx=0)
Definition: tensor.py:91
SampleMaskRCNN(const SampleMaskRCNNParams ¶ms)
Definition: sampleUffMaskRCNN.cpp:316
void write(const int *hOutputData, int actualOutputSequenceLength, int actualInputSequenceLength) override
write the generated sequence
Definition: bleuScoreWriter.cpp:163
def __init__(self, path)
Definition: backend/onnx/loader.py:25
constexpr int32_t EnumMax< TopKOperation >()
Maximum number of elements in TopKOperation enum.
Definition: NvInfer.h:4063
all it does is to evaluate BLEU score
Definition: bleuScoreWriter.h:34
Arguments argsToArgumentsMap(int argc, char *argv[])
Definition: sampleOptions.cpp:387
Class to handle library allocated memory that is accessible to the user.
Definition: NvInferRuntime.h:95
CodeTypeSSD
The type of encoding used for decoding the bounding boxes and loc_data.
Definition: NvInferPluginUtils.h:155
size_t mAnchorRatioNum
Definition: proposalPlugin.h:122
def super_verbose(self, message, mode=LogMode.EACH)
Definition: Polygraphy/polygraphy/logger/logger.py:254
Definition: sampleOptions.h:106
void cuErrCheck_(CUresult stat, const CUDADriverWrapper &wrap, const char *file, int line)
Definition: cudaDriverWrapper.h:88
def test_save_vectors(self, verbose)
Definition: quant_rnn_test.py:515
def forward(self, input)
Definition: quant_pooling.py:124
void configureWithFormat(const Dims *, int32_t, const Dims *, int32_t, DataType, PluginFormat, int32_t)
Derived classes should not implement this.
Definition: NvInferRuntimeCommon.h:700
std::map< std::string, std::string > mInOut
Input and output mapping of the network.
Definition: sampleINT8API.cpp:108
def infer(self, inputs, input_lengths)
Definition: demo/Tacotron2/tacotron2/model.py:667
virtual ~IPlugin()
Definition: NvInferRuntime.h:212
CalibrationAlgoType getAlgorithm() override
Signal that this is the entropy calibrator 2.
Definition: NvInfer.h:6084
def backward(ctx, grad_outputs)
Definition: tensor_quant.py:312
encoder
Definition: demo/Tacotron2/tacotron2/model.py:597
bool mAcrossSpatial
Definition: normalizePlugin.h:126
Definition: backend/trt_legacy.py:80
static const int MAX_LEVEL
Definition: tlt_mrcnn_config.h:61
Definition: batchedNMSPlugin.h:121
float latency() const
Definition: sampleReporting.h:59
std::string getInfo() override
get the textual description of the component
Definition: limitedSamplesDataReader.cpp:47
p_attention_dropout
Definition: demo/Tacotron2/tacotron2/model.py:250
float in
Definition: sampleReporting.h:53
nvinfer1::IPluginV2 * createPlugin(const char *name, const nvinfer1::PluginFieldCollection *fc) override
Return a plugin object.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:290
layerwise
Definition: backend/trt_legacy.py:223
int64_t mMaxWorkspaceSize
Definition: sampleConfig.h:46
const char * getPluginNamespace() const override
Return the namespace of the plugin creator object.
Definition: qkvToContextInt8InterleavedPlugin.cpp:358
Dims mInputDims
Definition: resizeNearestPlugin.h:91
void serialize(void *buffer) const override
Serialize the layer.
Definition: reorgPlugin.cpp:78
expr remquo(half x, half y, int *quo)
Remainder of division.
Definition: ieee_half.h:2264
void destroy() override
Destroy the plugin object.
Definition: resizeNearestPlugin.cpp:121
def convert_by_vocab(vocab, items)
Definition: tokenization.py:135
std::shared_ptr< std::istream > mInput
Definition: textReader.h:45
std::string mPluginName
Definition: batchedNMSPlugin.h:136
def __init__(self, kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=True, divisor_override=None, **kwargs)
Definition: quant_pooling.py:91
n_frames_per_step
Definition: tacotron2/data_function.py:87
bool build()
Creates the network, configures the builder and creates the network engine.
Definition: sampleUffPluginV2Ext.cpp:202
void * packed_mask_ptr
Definition: fused_multihead_attention_v2.h:30
def to_variable(self, np.dtype dtype=None, Sequence[Union[int, str]] shape=[])
Definition: tensor.py:74
def test_unsigned(self, verbose)
Definition: calibrator_test.py:157
uff_loader
Definition: backend/trt_legacy.py:82
Definition: backend/common/loader.py:1
unsigned char fused_multihead_attention_int8_128_64_kernel_sm80_cu_o[]
Definition: fused_multihead_attention_int8_128_64_kernel.sm80.cpp:20
bert::WeightsWithOwnership mPosEmb
Definition: embLayerNormVarSeqlenPlugin.h:101
virtual int32_t getNbBindings() const noexcept=0
Get the number of binding indices.
LSTMEncoder(ComponentWeights::ptr weights)
Definition: lstmEncoder.cpp:26
bool build()
Function builds the network engine.
Definition: sampleMNISTAPI.cpp:127
void destroy() override
Destroy the plugin object.
Definition: multilevelCropAndResizePlugin.cpp:115
__name__
Definition: graph.py:119
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: batchTilePlugin.cpp:68
SampleINT8Params mParams
The parameters for the sample.
Definition: sampleINT8.cpp:94
@ kLINEAR
Perform the normal matrix multiplication in the first recurrent layer.
def wrapper()
Definition: cuda.py:84
TrtCudaGraph & operator=(const TrtCudaGraph &)=delete
virtual int32_t getNbOutputs() const =0
Get the number of outputs from the layer.
float * getFileBatch()
Definition: BatchStream.h:285
std::vector< float > mLabels
Labels for the batch.
Definition: BatchStream.h:380
const std::string FCW_NAME
Definition: sampleCharRNN.cpp:77
std::string getInfo() override
get the textual description of the component
Definition: lstmEncoder.cpp:114
def toposort_linear_graph()
Definition: test_ir.py:376
onnx.TensorProto export_tensor_proto(Constant tensor)
Definition: onnx_exporter.py:32
const char * name
Definition: getopt.h:76
static const nvinfer1::Dims3 IMAGE_SHAPE
Definition: tlt_mrcnn_config.h:27
std::string mNamespace
Definition: plugin.h:57
void setPluginNamespace(const char *libNamespace) override
Set the namespace used to lookup and create plugins in the network.
Definition: caffeParser.h:56
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:24
_if_clip
Definition: tensor_quantizer.py:85
int dlaCore
Definition: sampleINT8API.cpp:59
int DLACore
Definition: sampleOptions.h:137
int mPreNMSTopK
Definition: proposalLayerPlugin.h:135
bool load
Definition: sampleOptions.h:118
void query(bool skipTransfers)
Definition: sampleInference.cpp:303
std::map< int32_t, std::vector< std::pair< int32_t, float > > > userToExpectedItemProbMap
Definition: sampleMovieLens.cpp:74
node
Definition: graph.py:264
Definition: sampleInference.h:33
~LogStreamConsumerBuffer()
Definition: pluginLogging.h:47
int initialize() override
Initialize the layer for execution.
Definition: specialSlicePlugin.cpp:127
void configurePlugin(const nvinfer1::DynamicPluginTensorDesc *in, int nbInputs, const nvinfer1::DynamicPluginTensorDesc *out, int nbOutputs) override
Definition: embLayerNormVarSeqlenPlugin.cpp:214
Definition: data_processing.py:1
num_attention_heads
Definition: quant_bert.py:59
bool interleaved
Definition: fused_multihead_attention_v2.h:68
def _run_split_on_punc(self, text)
Definition: tokenization.py:261
@ kFLOAT64
FP64 field type.
def __init__(self, obj=None, plugins=None)
Definition: backend/trt/loader.py:28
nvinfer1::DataType mModelDtype
Definition: parserOnnxConfig.h:47
def normalize_numbers(text)
Definition: numbers.py:79
tf32
Definition: backend/trt/loader.py:322
virtual const IDimensionExpr * constant(int32_t value)=0
Return pointer to IDimensionExp for given value.
half & operator--()
Prefix decrement.
Definition: samples/common/half.h:1530
size_t mInputWidth
Definition: cropAndResizePlugin.h:101
virtual void setModelDtype(const nvinfer1::DataType modelDtype)
Set the Model Data Type.
Definition: parserOnnxConfig.h:77
def test_set_severity(self, sev)
Definition: onnx/test_loader.py:31
weight_ih
Definition: modules/quant_rnn.py:390
def test_override_rv(self)
Definition: test_func.py:20
def get_input_metadata(self)
Definition: backend/onnxrt/runner.py:59
linear_projection
Definition: demo/Tacotron2/tacotron2/model.py:271
Definition: tools/Polygraphy/polygraphy/backend/onnx/__init__.py:1
T read(const char *&buffer)
Definition: plugin.h:87
T half2float(uint16 value)
Convert half-precision to floating point.
Definition: ieee_half.h:946
nvinfer1::Weights mKernelWeights
Definition: slpProjection.h:52
expr erf(half arg)
Error function.
Definition: ieee_half.h:2519
def unpack_args(args, num)
Definition: Polygraphy/polygraphy/util/misc.py:115
@ kCHAR
char field type. String for length>1.
TrtCudaGraph mGraph
Definition: sampleInference.cpp:459
Definition: qkvToContextInt8InterleavedPlugin.h:103
Definition: calibrator_test.py:140
nvinfer1::Dims mPredictionInputDims
The dimensions of the input of the MNIST model.
Definition: sampleDynamicReshape.cpp:80
int Verbosity
Defines Verbosity level.
Definition: NvOnnxConfig.h:55
const char * getEngineFileName() const
Definition: sampleConfig.h:129
def addReference(row, reference)
Definition: profiler.py:108
int iH
Definition: coordConvACPlugin.h:86
int numPriors
Definition: batchedNMSPlugin.h:115
const char * getPluginVersion() const override
Return the plugin version.
Definition: multilevelProposeROIPlugin.cpp:59
def load_and_setup_model(model_name, parser, checkpoint, fp16_run, cpu_run, forward_is_infer=False)
Definition: Tacotron2/inference.py:100
def get_text(self, text)
Definition: tacotron2/data_function.py:72
@ kPASSED
The test passed.
int getW(const Dims &d)
Definition: samples/common/common.h:891
Definition: quant_pooling_test.py:287
virtual IBinaryProtoBlob * parseBinaryProto(const char *fileName)=0
Parse and extract data stored in binaryproto file.
def __init__(self, Sequence[Node] nodes=None, Sequence[Tensor] inputs=None, Sequence[Tensor] outputs=None, name=None, doc_string=None, opset=None)
Definition: graph.py:105
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: normalizePlugin.cpp:177
static half_float::half round_error()
Maximum rounding error.
Definition: samples/common/half.h:4240
Definition: functional_test.py:35
def test_export_variable_tensor(self)
Definition: test_exporters.py:63
std::ostream & mErr
Definition: sampleEngines.cpp:188
IPluginV2Ext * deserializePlugin(const char *name, const void *data, size_t length) override
Called during deserialization of plugin layer.
Definition: resizeNearestPlugin.cpp:77
void reportTiming(int batchIndex, int batchSize)
Definition: bert_infer.h:305
def convert_tokens_to_ids(vocab, tokens)
Definition: tokenization.py:143
Interface implemented by application for selecting and reporting algorithms of a layer provided by th...
Definition: NvInfer.h:6305
Definition: deviceBuffer.h:27
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: cropAndResizePlugin.cpp:219
Node import_node(onnx.NodeProto onnx_node, "OrderedDict[str, Tensor]" tensor_map, "OrderedDict[str, Tensor]" subgraph_tensor_map)
Definition: onnx_importer.py:89
int _x_stride
Definition: split.h:43
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: normalizePlugin.cpp:171
std::string mNamespace
Definition: groupNormalizationPlugin.h:96
virtual void setModelFileName(const char *onnxFilename)=0
Set the Model File Name.
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: multilevelProposeROIPlugin.cpp:443
def __setattr__(self, name, value)
Definition: node.py:85
static void reportTestResult(const TestAtom &testAtom, TestResult result)
method that implements logging test results
Definition: pluginLogging.h:439
Definition: gridAnchorPlugin.h:134
def test_against_unquantized(self)
Definition: quant_conv_test.py:433
bool help
Definition: argsParser.h:83
int mBackgroundLabel
Definition: generateDetectionPlugin.h:94
def test_set_outputs_updates_old_outputs(self)
Definition: test_ir.py:61
def _load_from_state_dict(self, state_dict, prefix, *args, **kwargs)
Definition: tensor_quantizer.py:379
half & operator=(detail::expr rhs)
Assignment operator.
Definition: samples/common/half.h:1431
bool build()
Builds the network engine.
Definition: samplePlugin.cpp:127
int workspace
Definition: sampleOptions.h:109
def RNNReLUCell(input, hidden, w_ih, w_hh, b_ih=None, b_hh=None)
Definition: _functions/quant_rnn.py:36
def make_iterable(obj)
Definition: func.py:21
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: gridAnchorPlugin.cpp:309
void destroy() noexcept override
Destroy the allocated memory.
Definition: samples/common/common.h:343
def binding(self)
Definition: perf_varseqlen.py:32
int C
Definition: normalizePlugin.h:101
std::map< std::string, nvinfer1::Weights > mWeightMap
Definition: sampleCharRNN.cpp:185
The SampleMovieLensParams structure groups the additional parameters required by the MovieLens sample...
Definition: sampleMovieLens.cpp:62
std::string modelFileName
Definition: sampleINT8API.cpp:651
_unsigned
Definition: tools/pytorch-quantization/pytorch_quantization/calib/calibrator.py:34
def __init__(self, network, outputs=None, exclude_outputs=None)
Definition: backend/trt/loader.py:184
Verbosity getVerbosityLevel() const
Get the Verbosity Level.
Definition: sampleConfig.h:113
Definition: tools/pytorch-quantization/pytorch_quantization/nn/_functions/__init__.py:1
const char * getPluginName() const override
Return the plugin name.
Definition: detectionLayerPlugin.cpp:47
@ kSHIFT
shift part of IScaleLayer
nvinfer1::Dims mInputDims
The dimensions of the input to the network.
Definition: sampleINT8.cpp:96
def test_setitem(self, field_names)
Definition: test_ir.py:309
def refFeatures(names)
Definition: profiler.py:89
std::vector< nvinfer1::Dims > getStateSizes() override
get the sizes (vector of them) of the hidden state vectors
Definition: lstmDecoder.cpp:133
_scale_amax
Definition: tensor_quantizer.py:77
nvinfer1::Dims mOutputDims
The dimensions of the output to the network.
Definition: sampleINT8API.cpp:112
const char * getPluginName() const override
Return the plugin name.
Definition: priorBoxPlugin.cpp:368
Definition: NvInferRuntime.h:417
virtual size_t getWorkspaceSize(int32_t maxBatchSize) const =0
Find the workspace size required by the layer.
MatrixOperation
Enumerates the operations that may be performed on a tensor by IMatrixMultiplyLayer before multiplica...
Definition: NvInfer.h:4132
virtual ~SampleCharRNNBase()=default
float nmsThreshold
Definition: NvInferPluginUtils.h:185
def forward(self, input)
Definition: quant_pooling.py:85
NMSPluginCreator()
Definition: nmsPlugin.cpp:260
int mQ
Definition: sampleUffPluginV2Ext.cpp:177
int mKeepTopK
Definition: proposalLayerPlugin.h:136
constexpr int32_t EnumMax< NetworkDefinitionCreationFlag >()
Maximum number of elements in NetworkDefinitionCreationFlag enum.
Definition: NvInfer.h:6916
def _get_used_node_ids(self)
Definition: graph.py:182
Definition: checkMacrosPlugin.h:119
void dump(std::ostream &os, const std::string separator=" ") const
Definition: sampleUtils.h:268
Entropy calibrator.
Definition: NvInfer.h:6063
ILogger * gLogger
Definition: checkMacrosPlugin.cpp:28
std::vector< nvinfer1::Weights > getAllWeights(const std::string &layerName)
Definition: caffeWeightFactory.cpp:131
std::unique_ptr< trtcaffe::NetParameter > mRef
Definition: caffeWeightFactory.h:59
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: batchedNMSPlugin.cpp:248
IPluginV2DynamicExt * clone() const =0
Clone the plugin object.
size_t size(const std::string &tensorName) const
Returns the size of the host and device buffers that correspond to tensorName.
Definition: buffers.h:313
Definition: maskRCNNKernels.h:255
Wrapper for unary half-precision functions needing specialization for individual argument types.
Definition: ieee_half.h:1050
void serialize(void *buffer) const override
Serialize the layer.
Definition: multilevelCropAndResizePlugin.cpp:217
gate_threshold
Definition: demo/Tacotron2/tacotron2/model.py:249
Definition: caffeParser.h:32
Definition: NvInferPluginUtils.h:58
def copy_state_and_quantize_fused(dst, src, num_bits)
Definition: tools/pytorch-quantization/tests/utils.py:52
def test_combine_dicts_second_overwrites_first()
Definition: onnx-graphsurgeon/tests/test_util.py:20
def parse_args()
Definition: inference_varseqlen.py:39
size_t mSize
Definition: buffers.h:172
const Dims DataType
Definition: NvInferRuntimeCommon.h:792
def test_equals(self)
Definition: test_ir.py:52
void setPluginNamespace(const char *libNamespace) override
Set the namespace that this plugin object belongs to.
Definition: sampleUffPluginV2Ext.cpp:558
objs
Definition: Polygraphy/polygraphy/util/misc.py:442
constexpr size_t threadsPerCta384
Definition: embLayerNormVarSeqlenPlugin.cpp:34
def forward(self, input_data)
Definition: stft.py:139
def i(self, tensor_idx=0, producer_idx=0)
Definition: node.py:44
def enable_quant(self)
Definition: tensor_quantizer.py:186
use_int8
Definition: builder_varseqlen.py:99
half round(half arg)
Nearest integer.
Definition: ieee_half.h:2572
Plugin factory used to configure plugins.
Definition: NvUffParser.h:98
virtual int getStartSequenceId()=0
int inputWidth
Definition: sampleUffFasterRCNN.cpp:70
std::string exportOutput
Definition: sampleOptions.h:178
SampleUniquePtr< nvinfer1::IExecutionContext > mPredictionContext
Definition: sampleDynamicReshape.cpp:86
@ kNCHW
Deprecated name of kLINEAR, provided for backwards compatibility and will be removed in TensorRT 8....
def all_weights(self)
Definition: modules/quant_rnn.py:265
def test_clear(self, field_names)
Definition: test_ir.py:286
dropout_state
Definition: modules/quant_rnn.py:52
virtual void setDimensions(Dims dimensions)=0
Set the dimensions of a tensor.
bert::WeightsWithOwnership mTokEmb
Definition: embLayerNormVarSeqlenPlugin.h:100
bert::cuda_unique_ptr< void > mTokEmbDev
Definition: embLayerNormVarSeqlenPlugin.h:91
int64_t volume(const nvinfer1::Dims &d)
Definition: parserUtils.h:47
float findMedian(const std::vector< InferenceTime > &timings, const T &toFloat)
Find median in a sorted sequence of timings.
Definition: sampleReporting.cpp:55
@ kFLOAT16
FP16 field type.
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: lReluPlugin.cpp:150
PoolingType
The type of pooling to perform in a pooling layer.
Definition: NvInfer.h:1706
std::string mPluginNamespace
Definition: reorgPlugin.h:89
def __call__(self)
Definition: backend/tf/loader.py:290
void insertShapesInference(std::unordered_map< std::string, std::vector< int >> &shapes, const std::string &name, const std::vector< int > &dims)
Definition: sampleOptions.cpp:223
const char * getPluginType() const override
Return the plugin type.
Definition: proposalPlugin.cpp:150
static PluginFieldCollection mFC
Definition: cropAndResizePlugin.h:123
IPluginV2DynamicExt * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: groupNormalizationPlugin.cpp:309
virtual nvinfer1::DataType getDataType()=0
def test_fake_quant_per_channel_bias(self)
Definition: quant_conv_test.py:191
nvinfer1::ILayer * parseAbsVal(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &, BlobNameToTensor &tensors)
Definition: parseAbsVal.cpp:23
path
Definition: backend/tf/loader.py:119
Definition: backend/tf/loader.py:111
def set_output_range(layer, maxval, out_idx=0)
Definition: builder_varseqlen.py:117
The SampleMovieLens class implements the MovieLens sample.
Definition: sampleMovieLens.cpp:83
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: reorgPlugin.cpp:121
def forward(self, input)
Definition: quant_instancenorm.py:60
constexpr size_t xmmasM256
Definition: embLayerNormVarSeqlenPlugin.cpp:38
nvinfer1::ILayer * parseBNLL(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &, BlobNameToTensor &tensors)
Definition: parseBNLL.cpp:23
unsigned int shift_coeff
Definition: reducedMathPlugin.h:97
RPROIParams params
Definition: nvFasterRCNNPlugin.h:122
def severity(self)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:112
def mergeRow(reference, profile, diff)
Definition: profiler.py:121
Dims loadPGMFile(const std::string &fileName)
Loads a PGM file into mInput and returns the dimensions of the loaded image.
Definition: sampleDynamicReshape.cpp:371
virtual bool parse(void const *serialized_onnx_model, size_t serialized_onnx_model_size)=0
Parse a serialized ONNX model into the TensorRT network.
def test_can_get_shape(self)
Definition: test_ir.py:127
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleSSD.cpp:222
def test_layer_with_attrs(self)
Definition: test_ir.py:495
onnx.GraphProto export_graph(Graph graph, do_type_check=True)
Definition: onnx_exporter.py:68
_scale
Definition: tensor_quantizer.py:82
void serialize(void *buffer) const override
Serialize the layer.
Definition: specialSlicePlugin.cpp:109
@ kSUM
Sum of the two elements.
bool constructNetwork(SampleUniquePtr< nvinfer1::IBuilder > &builder, SampleUniquePtr< nvinfer1::INetworkDefinition > &network, SampleUniquePtr< nvinfer1::IBuilderConfig > &config, SampleUniquePtr< nvonnxparser::IParser > &parser)
Parses an ONNX model for MNIST and creates a TensorRT network.
Definition: sampleOnnxMnistCoordConvAC.cpp:171
@ kCONCATENATE
Output value is concatenation of values of tensor for each iteration, in forward order.
virtual int getBatchesRead() const =0
cudnnPoolingDescriptor_t mPoolingDesc
Definition: sampleUffPluginV2Ext.cpp:633
void destroy() override
Destroy the plugin object.
Definition: batchedNMSPlugin.cpp:380
vocab
Definition: tokenization.py:334
MultilevelProposeROIPluginCreator()
Definition: multilevelProposeROIPlugin.cpp:41
std::string mPluginName
Definition: gridAnchorPlugin.h:86
Descriptor for two-dimensional spatial data.
Definition: NvInfer.h:95
const char * DETECTIONLAYER_PLUGIN_VERSION
Definition: detectionLayerPlugin.cpp:28
A layer that represents the identity function.
Definition: NvInfer.h:4257
size_t mPreNmsTopN
Definition: proposalPlugin.h:117
virtual IParserError const * getError(int index) const =0
Get an error that occurred during prior calls to parse.
void destroy() override
Destroy the plugin object.
Definition: instanceNormalizationPlugin.cpp:249
Layer profile information.
Definition: sampleReporting.h:158
int mROICount
Definition: pyramidROIAlignPlugin.h:97
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: specialSlicePlugin.cpp:188
~SharedMemory()
Definition: sampleMovieLensMPS.cpp:126
bool runInInt8
Definition: argsParser.h:81
int C
Definition: nvFasterRCNNPlugin.h:99
nvinfer1::DataType getDataType() override
Definition: binaryProtoBlob.h:41
nvinfer1::ILayer * parsePooling(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &, BlobNameToTensor &tensors)
Definition: parsePooling.cpp:23
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: generateDetectionPlugin.cpp:255
unsigned char fused_multihead_attention_v2_int8_384_64_kernel_sm75_cubin[]
Definition: fused_multihead_attention_v2_int8_384_64_kernel.sm75.cpp:19
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: cropAndResizePlugin.cpp:214
def postprocess(run_results, postprocess_func)
Definition: comparator/comparator.py:188
def forward(self, signal)
Definition: layers.py:53
def check_onnx_parser_errors(parser)
Definition: trt/util.py:34
float * getFileLabels()
Definition: BatchStream.h:290
uint32_t TacticSources
Represents a collection of one or more TacticSource values combine using bitwise-OR operations.
Definition: NvInfer.h:6457
conv
Definition: layers.py:43
unsigned int fused_multihead_attention_v2_int8_128_64_kernel_sm75_cubin_len
Definition: fused_multihead_attention_v2_int8_128_64_kernel.sm75.cpp:15845
bool isDebug()
Definition: samples/common/common.h:370
The DetectionOutput plugin layer generates the detection output based on location and confidence pred...
Definition: NvInferPluginUtils.h:181
constexpr int defaultBatch
Definition: sampleOptions.h:44
def test_set_severity(self, sev)
Definition: trt/test_runner.py:31
std::shared_ptr< std::ostream > mOut
Definition: debugUtil.h:68
virtual const char * getTextFileName() const
Returns the File Name of the Network Description as a Text File.
Definition: parserOnnxConfig.h:112
virtual int32_t enqueue(int32_t batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream)=0
Execute the layer.
void destroy() override
Destroy the plugin object.
Definition: normalizePlugin.cpp:239
Definition: quant_pooling.py:141
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: proposalLayerPlugin.cpp:248
std::shared_ptr< PinnedHostBuffer< T > > ptr
Definition: pinnedHostBuffer.h:35
DataType
The type of weights and tensors.
Definition: NvInferRuntimeCommon.h:136
virtual ~IRefitter()
Definition: NvInferRuntime.h:885
Definition: builder_varseqlen.py:89
Definition: NvInferRuntimeCommon.h:876
std::string uffFileName
The file name of the UFF model to use.
Definition: sampleUffFasterRCNN.cpp:61
EmbLayerNormVarSeqlenPluginCreator()
Definition: embLayerNormVarSeqlenPlugin.cpp:419
Plugin class for user-implemented layers.
Definition: NvInferRuntimeCommon.h:400
std::string loadEngine
Definition: sampleUffFasterRCNN.cpp:86
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: pyramidROIAlignPlugin.cpp:127
virtual IErrorRecorder * getErrorRecorder() const =0
get the ErrorRecorder assigned to this interface.
void createEnqueueFunction(const InferenceOptions &inference, nvinfer1::IExecutionContext &context, Bindings &bindings)
Definition: sampleInference.cpp:434
@ kELU
Elu activation: x>=0 ? x : alpha * (exp(x) - 1).
n_early_every
Definition: demo/Tacotron2/waveglow/model.py:167
Definition: backend/trt/loader.py:27
bool constructNetwork(SampleUniquePtr< nvcaffeparser1::ICaffeParser > &parser, SampleUniquePtr< nvinfer1::INetworkDefinition > &network)
uses a Caffe parser to create the MNIST Network and marks the output layers.
Definition: sampleAlgorithmSelector.cpp:510
def _add_to_elem(self, elem)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:76
Severity getReportableSeverity() const
Definition: pluginLogging.h:391
def __str__(self)
Definition: backend/trt_legacy.py:184
Definition: modules/quant_rnn.py:336
bool mShouldLog
Definition: pluginLogging.h:180
interleaved
Definition: builder_varseqlen.py:101
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: gridAnchorPlugin.cpp:379
std::string imageFileName
Definition: sampleINT8API.cpp:65
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleCharRNN.cpp:809
def __init__(self, gpu_memory_fraction=None, allow_growth=None, use_xla=None)
Definition: backend/tf/loader.py:358
Definition: tacotron2/data_function.py:83
int d
Definition: fused_multihead_attention_v2.h:56
int getBatchSize() const override
Definition: BatchStream.h:274
int _axis
Definition: split.h:40
def test_no_quant(self)
Definition: quant_instancenorm_test.py:48
LogStreamConsumerBuffer mBuffer
Definition: samples/common/logging.h:120
bool confSigmoid
Definition: NvInferPluginUtils.h:188
bool readPerTensorDynamicRangeValues()
Populate per tensor dynamic range values.
Definition: sampleINT8API.cpp:184
int initialize() override
Initialize the layer for execution.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:223
Carries information about input or output of the algorithm. IAlgorithmIOInfo for all the input and ou...
Definition: NvInfer.h:6169
virtual bool isNetworkOutput() const =0
Whether the tensor is a network output.
def __init__(self, n_in_channels, n_mel_channels, n_layers, n_channels, kernel_size)
Definition: demo/Tacotron2/waveglow/model.py:89
std::string mNamespace
Definition: batchedNMSPlugin.h:71
int nmsMaxOut
The maximum number of detection post-NMS.
Definition: sampleFasterRCNN.cpp:48
int mUpdatesCount
Definition: sampleReporting.h:190
def parse_args()
Definition: BERT/inference.py:39
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: resizeNearestPlugin.cpp:258
int initialize() override
Initialize the layer for execution.
Definition: sampleUffPluginV2Ext.cpp:400
bool isInput
Definition: sampleUtils.h:217
@ kDIMS
nvinfer1::Dims field type.
def reduce_tensor(tensor, num_gpus)
Definition: train.py:151
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: normalizePlugin.cpp:107
def print_stats(measurements_all)
Definition: test_infer_trt.py:100
Data_type mDataType
Definition: fused_multihead_attention.h:277
void readLabelsFile(const std::string &labelsFilePath)
Definition: BatchStream.h:127
void generate_pyramid_anchors(const nvinfer1::Dims &image_size)
Definition: multilevelProposeROIPlugin.cpp:345
def convert_WN_1d_to_2d_(WN)
Definition: export_waveglow_onnx.py:66
nvinfer1::IPluginV2DynamicExt * clone() const override
Clone the plugin object.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:100
float computeEnd
Definition: sampleReporting.h:97
virtual ~IDimensionExpr()
Definition: NvInferRuntime.h:336
samplesCommon::CaffeSampleParams mParams
The parameters for the sample.
Definition: sampleAlgorithmSelector.cpp:359
ParserOnnxConfig()
Definition: parserOnnxConfig.h:52
do_lower_case
Definition: tokenization.py:224
def check_forward_input(self, input)
Definition: modules/quant_rnn.py:318
def test_unsigned_one_tensor(self, verbose)
Definition: calibrator_test.py:213
A LRN layer in a network definition.
Definition: NvInfer.h:1993
void setPluginNamespace(const char *libNamespace) override
Set the namespace that this plugin object belongs to.
Definition: plugin.h:47
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: debugUtil.cpp:81
void resizePPM(const PPM< T > &src, PPM< T > &dst, int target_height, int target_width, int channel)
Definition: sampleUffMaskRCNN.cpp:105
def test_unused_kwargs(self)
Definition: quant_conv_test.py:263
Definition: export_tacotron2_onnx.py:63
def evaluate(dataset, predictions, f1_acc)
Definition: evaluate-v1.1.py:71
Definition: tools/pytorch-quantization/pytorch_quantization/nn/__init__.py:1
void copyEmbeddingToInput(samplesCommon::BufferManager &buffers, const char &c)
Looks up the embedding tensor for a given char and copies it to input buffer.
Definition: sampleCharRNN.cpp:895
float x1
Definition: sampleUffMaskRCNN.cpp:54
bool mSmooth
Definition: bleuScoreWriter.h:57
def test_exclude_outputs_with_mark_layerwise(self, identity_identity_network)
Definition: trt/test_loader.py:137
def build_sensitivity_profile(model, criterion, data_loader_test)
Definition: classification_flow.py:421
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: sampleUffPluginV2Ext.cpp:411
static TFusedMHAKernelFactory< TFusedMHAKernelList > & Get()
Definition: fused_multihead_attention.h:313
Definition: tensor_quant.py:366
static expr atan(float arg)
Arc tangent implementation.
Definition: samples/common/half.h:1926
EnqueueExplicit(nvinfer1::IExecutionContext &context, void **buffers)
Definition: sampleInference.cpp:218
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: specialSlicePlugin.cpp:52
void fill(int binding)
Definition: sampleUtils.h:366
bool processInput(const samplesCommon::BufferManager &buffers, const float *data)
Reads the input and stores it in a managed buffer.
Definition: sampleINT8.cpp:369
def check_forward_hidden(self, input, hx, hidden_label='')
Definition: modules/quant_rnn.py:324
half fmin(half x, half y)
Minimum of half expressions.
Definition: ieee_half.h:2299
static size_t get_size_in_bytes(size_t n, Data_type dtype)
Definition: fused_multihead_attention.h:30
int oC
Definition: coordConvACPlugin.h:87
std::unordered_map< std::string, float > mPerTensorDynamicRangeMap
Mapping from tensor name to max absolute dynamic range values.
Definition: sampleINT8API.cpp:115
Definition: NvInfer.h:4616
SampleMaskRCNNParams mParams
Definition: sampleUffMaskRCNN.cpp:330
cntk_model
Definition: backend/cntk/runner.py:32
def to_yaml(self)
Definition: tensor_quant.py:200
void destroy() override
Destroy the plugin object.
Definition: pyramidROIAlignPlugin.cpp:102
TestAtom(bool started, const std::string &name, const std::string &cmdline)
Definition: samples/common/logging.h:274
uint64_t hashID(unsigned int s, bool interleaved, bool unroll) const
Definition: fused_multihead_attention_v2.h:433
RNNOperation
Enumerates the RNN operations that may be performed by an RNN layer.
Definition: NvInfer.h:2791
float y1
Definition: samples/common/common.h:669
def __enter__(self)
Definition: backend/base/runner.py:66
Definition: priorBoxPlugin.h:32
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleSSD.cpp:63
void setProtobufBufferSize(size_t size) override
Set buffer size for the parsing and storage of the learned model.
Definition: caffeParser.h:47
virtual void stop()
Definition: samples/common/common.h:785
cudnnTensorDescriptor_t _y_desc
Definition: instanceNormalizationPlugin.h:98
bool processInput(const samplesCommon::BufferManager &buffers)
Reads the input and mean data, preprocesses, and stores the result in a managed buffer.
Definition: sampleSSD.cpp:276
int calBatchSize
The calibration batch size.
Definition: sampleINT8.cpp:50
nvinfer1::DimsExprs getOutputDimensions(int outputIndex, const nvinfer1::DimsExprs *inputs, int nbInputs, nvinfer1::IExprBuilder &exprBuilder) override
Definition: embLayerNormVarSeqlenPlugin.cpp:131
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: nmsPlugin.cpp:142
int32_t batchSize
Number of inputs in a batch.
Definition: argsParser.h:38
const char * PROPOSALLAYER_PLUGIN_VERSION
Definition: proposalLayerPlugin.cpp:32
Definition: quant_conv_test.py:267
nvinfer1::ITensor * find(const char *name) const override
Given a blob name, returns a pointer to a ITensor object.
Definition: blobNameToTensor.h:36
std::array< TimePoint, 2 > EnqueueTimes
Definition: sampleInference.cpp:275
void skip(int skipCount) override
Definition: BatchStream.h:243
def test_unique_list(case)
Definition: test_misc.py:100
SharedMemory(const char *modelStreamFd)
Definition: sampleMovieLensMPS.cpp:121
gate_layer
Definition: demo/Tacotron2/tacotron2/model.py:275
size_t getWorkspaceSize(const nvinfer1::PluginTensorDesc *inputs, int nbInputs, const nvinfer1::PluginTensorDesc *outputs, int nbOutputs) const override
Definition: groupNormalizationPlugin.cpp:235
unsigned int fused_multihead_attention_fp16_384_64_kernel_sm75_cu_o_len
Definition: fused_multihead_attention_fp16_384_64_kernel.sm75.cpp:1819
def test_custom_outputs(self)
Definition: onnx/test_loader.py:60
def test_extend_can_modify_rv(self)
Definition: test_func.py:106
hidden_size
Definition: builder_varseqlen.py:94
run_metadata
Definition: backend/tf/runner.py:56
static const char * severityPrefix(Severity severity)
returns an appropriate string for prefixing a log message with the given severity
Definition: samples/common/logging.h:379
void constructNetwork(SampleUniquePtr< nvinfer1::IBuilder > &builder, SampleUniquePtr< nvinfer1::INetworkDefinition > &network, SampleUniquePtr< nvinfer1::IBuilderConfig > &config)
Create full model using the TensorRT network definition API and build the engine.
Definition: sampleCharRNN.cpp:761
int64_t implementation
Definition: sampleAlgorithmSelector.cpp:237
Layer that represents a padding operation.
Definition: NvInfer.h:3638
LogStreamConsumerBuffer(std::ostream &stream, const std::string &prefix, bool shouldLog)
Definition: samples/common/logging.h:37
int mFeatureCnt
Definition: multilevelProposeROIPlugin.h:97
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: multilevelCropAndResizePlugin.cpp:140
@ kYOLOREGION
YOLO Region Plugin.
obj
Definition: backend/trt/loader.py:48
CoordConvACPluginCreator()
Definition: coordConvACPlugin.cpp:178
Threads synchronization structure.
Definition: sampleInference.cpp:166
expr sqrt(half arg)
Square root.
Definition: ieee_half.h:2379
def test_uff_identity(fp16, tf32)
Definition: test_tensorrt_legacy.py:26
const char * getPluginName() const override
Return the plugin name.
Definition: proposalPlugin.cpp:334
Definition: logger.cpp:20
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: normalizePlugin.cpp:227
CUresult(* _cuLaunchCooperativeKernel)(CUfunction, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, CUstream, void **)
Definition: cudaDriverWrapper.h:81
unsigned char fused_multihead_attention_v2_fp16_64_64_kernel_sm80_cubin[]
Definition: fused_multihead_attention_v2_fp16_64_64_kernel.sm80.cpp:19
int enqueue(int batch_size, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: detectionLayerPlugin.cpp:254
def __init__(self, filter_length=800, hop_length=200, win_length=800, window='hann')
Definition: stft.py:44
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleOnnxMNIST.cpp:196
@ kCHW4
Four wide channel vectorized row major format.
def pytest_addoption(parser)
Definition: conftest.py:22
virtual void terminate() override
Release resources acquired during plugin layer initialization.
Definition: samples/opensource/samplePlugin/fcPlugin.h:146
virtual uint64_t hashID(const KernelMeta &kernelMeta) const
Definition: fused_multihead_attention_v2.h:438
Definition: base_exporter.py:19
max_input_chars_per_word
Definition: tokenization.py:336
static void serialize(void **buffer, std::vector< T > const &value)
Definition: serialize.hpp:95
~InferenceTrace()=default
const char * getPluginVersion() const override
Return the plugin version.
Definition: normalizePlugin.cpp:272
uint64_t mTopK
Definition: sampleConfig.h:57
void backtrack(int lastTimestepId, int sampleId, int lastTimestepRayId, int *hOutputData, int lastTimestepWriteId) const
Definition: beamSearchPolicy.cpp:153
def forward(self, audio, strength=0.1)
Definition: denoiser.py:47
void finalize() override
it is called right after inference ends
Definition: bleuScoreWriter.cpp:185
std::shared_ptr< Decoder > ptr
Definition: decoder.h:36
@ kTRIP_LIMIT
Loop Trip limit layer.
Definition: integration_test.py:43
void parse(Arguments &arguments) override
Definition: sampleOptions.cpp:708
half operator--(int)
Postfix decrement.
Definition: samples/common/half.h:1546
def copy_from(self, host_buffer, stream=None)
Definition: cuda.py:160
def main()
Definition: copyright-scan.py:165
def get_go_frame(self, memory)
Definition: demo/Tacotron2/tacotron2/model.py:286
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: sampleUffPluginV2Ext.cpp:552
const char * getPluginName() const override
Return the plugin name.
Definition: batchTilePlugin.cpp:189
bool mPrintLayercInfo
Definition: parserOnnxConfig.h:49
std::string referenceFileName
Definition: sampleINT8API.cpp:653
PyramidROIAlign(int pooled_size)
Definition: pyramidROIAlignPlugin.cpp:78
void addToModel(nvinfer1::INetworkDefinition *network, nvinfer1::ITensor *input, nvinfer1::ITensor **output) override
add the embedding vector calculation to the network
Definition: slpEmbedder.cpp:41
virtual std::vector< nvinfer1::Dims > getStateSizes()=0
get the sizes (vector of them) of the hidden state vectors
virtual bool getPrintLayerInfo() const
Get whether the layer information will be printed.
Definition: parserOnnxConfig.h:128
conv1
Definition: tools/pytorch-quantization/tests/fixtures/models.py:29
int randomSeed
Definition: demo/BERT/infer_c/common.h:34
int mInputChannelCount
Definition: slpAttention.h:49
CUresult cuModuleGetFunction(CUfunction *hfunc, CUmodule hmod, const char *name) const
Definition: cudaDriverWrapper.cpp:107
def default_comparisons(run_results)
Definition: comparator/comparator.py:208
bool constructNetwork(SampleUniquePtr< nvinfer1::IBuilder > &builder, SampleUniquePtr< nvinfer1::INetworkDefinition > &network, SampleUniquePtr< nvinfer1::IBuilderConfig > &config)
Uses the API to create the MLP Network.
Definition: sampleMLP.cpp:188
int numPriors
Definition: nmsPlugin.h:90
std::vector< std::string > Segment_t
Definition: bleuScoreWriter.cpp:32
def build_basic_graph()
Definition: test_ir.py:338
@ kFLOOR_DIV
Floor division of the first element by the second.
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: lReluPlugin.cpp:93
size_t mCapacity
Definition: buffers.h:172
window
Definition: stft.py:49
def lazy_write(contents, path, mode="wb")
Definition: Polygraphy/polygraphy/util/misc.py:295
GridAnchorPluginCreator()
Definition: gridAnchorPlugin.cpp:497
Definition: optim_helper_test.py:103
InputDataFormat
Definition: sampleConfig.h:29
std::string mNamespace
Definition: batchTilePlugin.h:111
float mIOUThreshold
Definition: detectionLayerPlugin.h:97
static PluginFieldCollection mFC
Definition: resizeNearestPlugin.h:114
Definition: clip_test.py:36
IPluginV2Ext * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: regionPlugin.cpp:528
def quant_lenet()
Definition: tools/pytorch-quantization/tests/fixtures/models.py:67
convolutions
Definition: demo/Tacotron2/tacotron2/model.py:129
def test_simple_run(self)
Definition: tensor_quantizer_test.py:38
size_t mDepth
Definition: cropAndResizePlugin.h:101
std::vector< nvinfer1::PluginField > parseDetectionOutputParam(const trtcaffe::LayerParameter &msg, CaffeWeightFactory &weightFactory, BlobNameToTensor &tensors)
Definition: caffeParser.cpp:158
Definition: sampleEngines.cpp:157
float mFailurePercentage
Definition: sampleConfig.h:58
nvinfer1::ILayer * parsePower(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &weightFactory, BlobNameToTensor &tensors)
Definition: parsePower.cpp:23
Definition: trt/test_runner.py:29
std::string saveEngine
Definition: argsParser.h:87
void operator()(void *ptr)
Definition: sampleDevice.h:299
use_strict
Definition: builder.py:100
half modf(half arg, half *iptr)
Extract integer and fractional parts.
Definition: ieee_half.h:2643
LayerType
The type values of layer classes.
Definition: NvInfer.h:444
const char * OUTPUT_BLOB_NAME
Definition: sampleCharRNN.cpp:92
Definition: tools/Polygraphy/polygraphy/util/__init__.py:1
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace of the plugin creator based on the plugin library it belongs to.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:339
void addBBoxPPM(PPM< uint8_t > &ppm, const BBoxInfo &box, const PPM< uint8_t > &resized_mask)
Definition: sampleUffMaskRCNN.cpp:265
virtual ~LikelihoodCombinationOperator()=default
void splitInsertKeyValue(const std::vector< std::string > &kvList, T &map)
Definition: sampleOptions.cpp:163
Definition: test_api.py:25
SampleCharRNNv2(SampleCharRNNParams params)
Definition: sampleCharRNN.cpp:226
ModelFormat format
Definition: sampleOptions.h:76
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: nmsPlugin.cpp:196
expr acos(half arg)
Arc cosine function.
Definition: ieee_half.h:2445
Definition: backend/onnx/loader.py:49
Definition: backend/onnx/loader.py:178
def __init__(self, graph, max_workspace_size=None, fp16=None, int8=None, max_batch_size=None, is_dynamic_op=False, minimum_segment_size=None)
Definition: backend/tf/loader.py:213
nvinfer1::IPluginV2DynamicExt * clone() const override
Clone the plugin object.
Definition: embLayerNormVarSeqlenPlugin.cpp:121
static std::vector< PluginField > mPluginAttributes
Definition: specialSlicePlugin.h:118
The CaffeSampleParams structure groups the additional parameters required by networks that use caffe.
Definition: argsParser.h:51
def last_inference_time(self)
Definition: backend/base/runner.py:51
Managed CUDA stream.
Definition: sampleDevice.h:57
MultiStream mStream
Definition: sampleInference.cpp:467
virtual void setProtobufBufferSize(size_t size)=0
Set buffer size for the parsing and storage of the learned model.
friend std::istream & operator>>(std::istream &input, Vocabulary &value)
Definition: vocabulary.cpp:61
bool verbose
Definition: sampleMovieLens.cpp:529
sem_t * mSemEngine
Definition: sampleMovieLensMPS.cpp:115
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleUffFasterRCNN.cpp:125
int32_t nbDims
The number of dimensions.
Definition: NvInferRuntime.h:378
void bboxTransformInvAndClip(const float *rois, const float *deltas, float *predBBoxes, const float *imInfo, const int N, const int nmsMaxOut, const int numCls)
Performs inverse bounding box transform and clipping.
Definition: sampleFasterRCNN.cpp:384
const char * getPluginVersion() const override
Return the plugin version.
Definition: generateDetectionPlugin.cpp:170
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: proposalLayerPlugin.cpp:175
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: batchTilePlugin.cpp:199
constexpr const char * GROUP_NORM_NAME
Definition: groupNormalizationPlugin.cpp:38
std::vector< std::string > dataDirs
Directory paths where sample data files are stored.
Definition: argsParser.h:42
std::shared_ptr< Component > ptr
Definition: component.h:33
represents the core of attention mechanism
Definition: alignment.h:32
void setPluginNamespace(const char *libNamespace) override
Set the namespace that this plugin object belongs to.
Definition: resizeNearestPlugin.cpp:182
Definition: demo/Tacotron2/waveglow/model.py:82
void setOutputFileName(const char *outputFilename)
Definition: sampleConfig.h:263
const char * REGION_PLUGIN_VERSION
Definition: regionPlugin.cpp:27
virtual ~IInt8EntropyCalibrator()
Definition: NvInfer.h:6071
def __init__(self)
Definition: tensor.py:28
def question_features(tokens, question)
Definition: BERT/inference.py:114
virtual const char * getPluginNamespace() const =0
Return the namespace of the plugin object.
Definition: trt/test_loader.py:62
def float_to_hex(f)
Definition: convert_weights.py:58
static std::vector< PluginField > mPluginAttributes
Definition: lReluPlugin.h:96
@ kPRELU
YOLO PReLU Plugin.
cache
Definition: comparator/data_loader.py:201
def receive_on_queue(queue, timeout=None)
Definition: Polygraphy/polygraphy/util/misc.py:243
@ kTF32
Allow (but not require) computations on tensors of type DataType::kFLOAT to use TF32.
def test_clip_gradient(self)
Definition: tensor_quant_test.py:249
void readInputSample(std::ifstream &file, OutputParams &outParams, std::string line)
Extracts needed dataset values for a single user in the MovieLens, dataset .csv file,...
Definition: sampleMovieLens.cpp:374
float enq
Definition: sampleReporting.h:52
Semaphore(const char *semName)
Definition: sampleMovieLensMPS.cpp:83
const char * GENERATEDETECTION_PLUGIN_NAME
Definition: generateDetectionPlugin.cpp:32
def __call__(self)
Definition: backend/trt/loader.py:472
const char * getPluginVersion() const override
Return the plugin version.
Definition: batchTilePlugin.cpp:194
constexpr int32_t EnumMax< RNNGateType >()
Maximum number of elements in RNNGateType enum.
Definition: NvInfer.h:3214
Implements common functionality for Entropy calibrators.
Definition: EntropyCalibrator.h:28
WN
Definition: demo/Tacotron2/waveglow/model.py:169
def __enter__(self)
Definition: Polygraphy/polygraphy/logger/logger.py:49
Coupled host and device buffers.
Definition: sampleDevice.h:313
nvinfer1::DimsHW sortClassSampleIdxDims
Definition: maskRCNNKernels.h:102
~Attention() override=default
virtual float init() const =0
DataType type
The type of the weights.
Definition: NvInferRuntime.h:80
float enqStart
Definition: sampleReporting.h:92
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:69
Definition: lReluPlugin.h:32
int _nchan
Definition: instanceNormalizationPlugin.h:91
Definition: tf/test_loader.py:50
int32_t poolingW
Definition: NvInferPluginUtils.h:119
constexpr const char * INSTANCE_PLUGIN_NAME
Definition: instanceNormalizationPlugin.cpp:65
__host__ __device__ __forceinline__ reduced_divisor(unsigned _mul_coeff, unsigned _shift_coeff, int _y)
Definition: reducedMathPlugin.h:63
static expr multiplies(float x, float y)
Multiplication implementation.
Definition: samples/common/half.h:1607
nvinfer1::Dims mDims
Input dimensions.
Definition: BatchStream.h:385
static void serialize(void **buffer, T const &value)
Definition: serialize.hpp:51
void getInputOutputNames()
Populates input and output mapping of the network.
Definition: sampleINT8API.cpp:153
SampleCharRNNBindingNames bindingNames
Definition: sampleCharRNN.cpp:129
const char * file
Definition: checkMacrosPlugin.h:93
def infer_impl(self, feed_dict)
Definition: backend/trt/runner.py:152
Layer type for pluginV2.
Definition: NvInfer.h:3457
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: nmsPlugin.cpp:293
void operator()(void **ptr, size_t size)
Definition: sampleDevice.h:275
def reduce_amax(input, axis=None, keepdims=True)
Definition: reduce_amax.py:24
name
Definition: node.py:38
Definition: waveglow/loss_function.py:19
@ kUNIFORM
Identical coefficients across all elements of the tensor.
static std::vector< PluginField > mPluginAttributes
Definition: normalizePlugin.h:130
std::string mNameSpace
Definition: multilevelCropAndResizePlugin.h:102
Definition: Polygraphy/polygraphy/logger/logger.py:72
shape
Definition: cuda.py:122
float mScoreThreshold
Definition: detectionLayerPlugin.h:129
RNNDirection
Enumerates the RNN direction that may be performed by an RNN layer.
Definition: NvInfer.h:2813
unsigned char fused_multihead_attention_v2_int8_256_64_kernel_sm75_cubin[]
Definition: fused_multihead_attention_v2_int8_256_64_kernel.sm75.cpp:19
TrtUniquePtr< nvinfer1::ICudaEngine > getEngine(const ModelOptions &model, const BuildOptions &build, const SystemOptions &sys, std::ostream &err)
Create an engine from model or serialized file, and optionally save engine.
Definition: sampleEngines.cpp:682
BaseModelOptions baseModel
Definition: sampleOptions.h:96
fp16
Definition: backend/trt/loader.py:323
virtual ~IGpuAllocator()
Destructor declared virtual as general good practice for a class with virtual methods.
Definition: NvInferRuntimeCommon.h:1065
~Decoder() override=default
bool mPluginFactoryIsExt
Definition: caffeParser.h:89
const std::string RNNB_L0_NAME
Definition: sampleCharRNN.cpp:74
std::map< std::string, std::pair< nvinfer1::Dims, nvinfer1::Weights > > loadWeights(const std::string &file)
Loads weights from weights file.
Definition: sampleMLP.cpp:362
bert::WeightsWithOwnership mWordEmb
Definition: embLayerNormVarSeqlenPlugin.h:99
~LReluPluginCreator() override=default
Vocabulary::ptr mVocabulary
Definition: textWriter.h:50
std::string mNamespace
Definition: coordConvACPlugin.h:89
@ kLSTM
Four-gate LSTM network w/o peephole connections.
cudnnTensorDescriptor_t mDstDescriptor
Definition: samples/opensource/samplePlugin/fcPlugin.h:317
cublasHandle_t mCublas
Definition: flattenConcat.h:105
def __bool__(self)
Definition: comparator/struct.py:154
bool save
Definition: sampleOptions.h:117
nvinfer1::ITensor * hidden
Definition: sampleCharRNN.cpp:244
bool stepOnce(samplesCommon::BufferManager &buffers, SampleUniquePtr< nvinfer1::IExecutionContext > &context, cudaStream_t &stream)
Perform one time step of inference with the TensorRT execution context.
Definition: sampleCharRNN.cpp:908
@ kMAX
Maximum of the two operands.
@ kGREATER
Check if element in first tensor is greater than corresponding element in second tensor.
size_t getWorkspaceSize(int) const override
Definition: specialSlicePlugin.cpp:67
const char * MULTILEVELCROPANDRESIZE_PLUGIN_VERSION
Definition: multilevelCropAndResizePlugin.cpp:31
def test_calibrator_with_file_object_cache(self, identity_builder_network, mode)
Definition: test_calibrator.py:101
std::vector< float > mHostOutput
Definition: bert_infer.h:348
nvinfer1::IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: proposalPlugin.cpp:256
int enqueue(int batch_size, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: proposalLayerPlugin.cpp:308
TripLimit
Enum that describes kinds of trip limits.
Definition: NvInfer.h:4507
virtual float * getBatch()=0
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: flattenConcat.cpp:87
def __init__(self, data_loader, save_inputs_path=None)
Definition: comparator/data_loader.py:199
nodes
Definition: onnx_models.py:39
def _expand_decimal_point(m)
Definition: numbers.py:35
bias_ih
Definition: modules/quant_rnn.py:393
int _ny
Definition: split.h:42
half & operator/=(float rhs)
Arithmetic assignment.
Definition: samples/common/half.h:1515
def __getitem__(self, key)
Definition: backend/trt/loader.py:285
Definition: backend/trt/loader.py:458
nvinfer1::DataType mDataType
Definition: binaryProtoBlob.h:62
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: reorgPlugin.cpp:56
~TextWriter() override=default
const char * getOutputFileName() const
< specifies the file to save the results
Definition: sampleConfig.h:259
def __init__(self, **kwargs)
Definition: tools/pytorch-quantization/tests/fixtures/models.py:45
Definition: modules/quant_rnn.py:306
def float_to_hex(f)
Definition: dumpTFWts.py:51
def _quant(self, input)
Definition: quant_conv.py:252
Definition: NvInfer.h:4530
@ kXOR
Logical XOR of two elements.
int W
Definition: nvFasterRCNNPlugin.h:99
~RndInt8Calibrator()
Definition: sampleEngines.cpp:163
@ kSUB
Substract the second element from the first.
TrtUniquePtr< nvinfer1::ICudaEngine > engine
Definition: sampleInference.h:35
Definition: Polygraphy/polygraphy/logger/logger.py:77
std::string mNamespace
Definition: skipLayerNormInt8InterleavedPlugin.h:72
outputs
Definition: backend/trt/loader.py:204
Data_type
Definition: fused_multihead_attention_common.h:22
const char * getPluginType() const override
Return the plugin type.
Definition: proposalLayerPlugin.cpp:165
colors
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:104
Definition: tensor_quant_test.py:352
wraps another data reader and limits the number of samples to read
Definition: limitedSamplesDataReader.h:29
virtual ~IScaleLayer()
Definition: NvInfer.h:2164
virtual void setName(const char *name)=0
Sets the name of the network.
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: proposalLayerPlugin.cpp:61
static std::vector< nvinfer1::PluginField > mPluginAttributes
Definition: skipLayerNormInt8InterleavedPlugin.h:115
constexpr int32_t EnumMax< BuilderFlag >()
Maximum number of builder flags in BuilderFlag enum.
Definition: NvInfer.h:6402
const char * getPluginType() const override
Return the plugin type.
Definition: multilevelCropAndResizePlugin.cpp:130
def swap_inputs_outputs(decoder_inputs, decoder_outputs)
Definition: inference_trt.py:148
def test_one_tensor(self, verbose)
Definition: calibrator_test.py:142
_axis
Definition: tensor_quant.py:85
def setup_method(self)
Definition: test_ir.py:111
sess
Definition: backend/onnxrt/runner.py:44
const char * getPluginVersion() const override
Return the plugin version.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:214
@ kLEAKY_RELU
LeakyRelu activation: x>=0 ? x : alpha * x.
bool valueRequired
True if a value is needed for an option (i.e. -N 4, –foo bar)
Definition: getOptions.h:38
static expr sqrt(float arg)
Square root implementation.
Definition: samples/common/half.h:1839
bool isunordered(half x, half y)
Check if unordered.
Definition: ieee_half.h:2828
def forward(self, x)
Definition: tools/pytorch-quantization/tests/fixtures/models.py:34
nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType *inputType, int nbInputs) const override
Definition: coordConvACPlugin.cpp:160
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: qkvToContextInt8InterleavedPlugin.cpp:189
basic_tokenizer
Definition: tokenization.py:191
std::vector< void * > mTmpAllocs
Definition: caffeParser.h:84
def volume(obj)
Definition: Polygraphy/polygraphy/util/misc.py:161
static void help(std::ostream &out)
Definition: sampleOptions.cpp:1136
def test_device_buffer_memcpy_sync(self)
Definition: test_cuda.py:66
void constructNetwork(SampleUniquePtr< nvcaffeparser1::ICaffeParser > &parser, SampleUniquePtr< nvinfer1::INetworkDefinition > &network)
Parses a Caffe model for GoogleNet and creates a TensorRT network.
Definition: sampleGoogleNet.cpp:139
~ProposalLayerPluginCreator()
Definition: proposalLayerPlugin.h:121
outputs
Definition: node.py:41
def reset(self)
Definition: tools/pytorch-quantization/pytorch_quantization/calib/calibrator.py:44
void parse(Arguments &arguments) override
Definition: sampleOptions.cpp:494
bert::WeightsWithOwnership mGamma
Definition: embLayerNormVarSeqlenPlugin.h:98
const char * getPluginName() const override
Return the plugin name.
Definition: cropAndResizePlugin.cpp:253
FillOperation
Enumerates the tensor fill operations that may performed by a fill layer.
Definition: NvInfer.h:4731
calculates context vector from raw alignment scores and memory states
Definition: contextNMT.h:32
TypedHostMemory< float, DataType::kFLOAT > FloatMemory
Definition: samples/common/common.h:354
int32_t H
Definition: NvInferPluginUtils.h:147
def test_layerwise(self)
Definition: tf/test_loader.py:51
int mNumber
The number to classify.
Definition: sampleOnnxMnistCoordConvAC.cpp:79
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: gridAnchorPlugin.cpp:337
void serialize(void *buffer) const override
Serialize the layer.
Definition: priorBoxPlugin.cpp:192
device
Definition: backend/trt_legacy.py:182
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: multilevelCropAndResizePlugin.cpp:125
def __getattr__(self, name)
Definition: graph.py:129
IPluginV2Ext * deserializePlugin(const char *name, const void *data, size_t length) override
Called during deserialization of plugin layer.
Definition: generateDetectionPlugin.cpp:103
void setPluginNamespace(const char *libNamespace) override
Set the namespace of the plugin creator based on the plugin library it belongs to.
Definition: sampleUffPluginV2Ext.cpp:678
Definition: modules/quant_rnn.py:1
bert::WeightsWithOwnership mBeta
Definition: skipLayerNormInt8InterleavedPlugin.h:78
EventType
Definition: sampleInference.cpp:260
def main()
Definition: test_infer_trt.py:140
const char * getPluginNamespace() const
Return the namespace of the plugin creator object.
Definition: split.h:174
path
Definition: backend/trt/loader.py:502
def __init__(self, num_bits, axis, unsigned, track_amax=False)
Definition: max.py:37
void write(char *&buffer, const T &val)
Definition: plugin.h:79
bool_type< true > true_type
Definition: ieee_half.h:301
def __init__(self, parent_obj, field_name, initial)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:70
bool shareLocation
Definition: NvInferPluginUtils.h:249
Descriptor for data with one channel dimension and two spatial dimensions.
def pickle_load(path)
Definition: Polygraphy/polygraphy/util/misc.py:197
def refName(name)
Definition: profiler.py:83
def append(self, x)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:97
_network
Definition: backend/trt/loader.py:203
def test_input_fake_quant(self)
Definition: quant_pooling_test.py:207
TensorFormat format
Definition: sampleReformatFreeIO.cpp:187
def free(self)
Definition: perf_varseqlen.py:35
virtual int getInputDimensionSize()=0
get the upper bound for the possible values of indices
BlobNameToTensor * mBlobNameToTensor
Definition: caffeParser.h:85
The OnnxSampleParams structure groups the additional parameters required by networks that use ONNX.
Definition: argsParser.h:62
size_t mProtobufBufferSize
Definition: caffeParser.h:86
virtual ILayer * getLayer(int32_t index) const =0
Get the layer specified by the given index.
def get_input_metadata_from_engine(engine, start_binding, end_binding)
Definition: trt/util.py:373
Definition: cmudict.py:34
TrtCudaBuffer(TrtCudaBuffer &&rhs)
Definition: sampleDevice.h:223
Definition: functional.py:27
fc1
Definition: tools/pytorch-quantization/tests/fixtures/models.py:31
nvinfer1::Weights mKernelWeights
Definition: samples/opensource/samplePlugin/fcPlugin.h:309
unsigned char fused_multihead_attention_v2_int8_384_64_kernel_cubin[]
Definition: fused_multihead_attention_v2_int8_384_64_kernel.sm72.cpp:19
const char * getImageFileName() const
< set Image file name (PPM or ASCII)
Definition: sampleConfig.h:233
@ kDEVICE
Data stored on device.
int initialize() override
Initialize the layer for execution.
Definition: normalizePlugin.cpp:95
Definition: groupNormalizationPlugin.h:39
SkipLayerNormInterleavedPluginCreator()
Definition: skipLayerNormInt8InterleavedPlugin.cpp:269
Definition: quant_modules_test.py:32
int enqueue(int batch_size, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: multilevelProposeROIPlugin.cpp:391
~BLEUScoreWriter() override=default
Definition: quant_conv.py:309
@ kITERATOR
Loop Iterator layer.
int32_t imgH
Definition: NvInferPluginUtils.h:98
int mInputHeight
Definition: multilevelCropAndResizePlugin.h:99
def test_fold_constants(self)
Definition: test_ir.py:698
Definition: caffeWeightFactory.h:30
Main namespace for half precision functionality.
Definition: ieee_half.h:262
seed
Definition: comparator/data_loader.py:86
bool processInput(const samplesCommon::BufferManager &buffers, const std::string &inputTensorName, int inputFileIdx) const
Reads the input and mean data, preprocesses, and stores the result in a managed buffer.
Definition: sampleUffMNIST.cpp:196
nvinfer1::Weights getWeights(const trtcaffe::BlobProto &blobMsg, const std::string &layerName)
Definition: caffeWeightFactory.cpp:395
Definition: quant_instancenorm.py:1
std::unordered_map< std::string, std::string > inputs
Definition: sampleOptions.h:161
int * flag
Definition: getopt.h:78
std::string exportProfile
Definition: sampleOptions.h:179
~CUDADriverWrapper()
Definition: cudaDriverWrapper.cpp:66
Definition: generateDetectionPlugin.h:112
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: coordConvACPlugin.cpp:106
def register_callback()
Definition: tools/Polygraphy/polygraphy/backend/trt/__init__.py:7
def test_extend(self, field_names)
Definition: test_ir.py:237
Weights aspectRatios
Definition: priorBoxPlugin.h:94
virtual bool isOutputBroadcastAcrossBatch(int32_t outputIndex, const bool *inputIsBroadcasted, int32_t nbInputs) const =0
Return true if output tensor is broadcast across a batch.
@ kFAILED_INITIALIZATION
One, or more, of the components that TensorRT relies on did not initialize correctly.
Definition: reorgPlugin.h:92
const char * getPluginVersion() const override
Return the plugin version.
Definition: detectionLayerPlugin.cpp:153
def get_bindings_per_profile(engine)
Definition: trt/util.py:428
constexpr int32_t EnumMax< ResizeMode >()
Maximum number of elements in ResizeMode enum.
Definition: NvInfer.h:4339
def test_dim_param_converted_to_int_shape(self)
Definition: onnxrt/test_runner.py:51
def test_generate_name(self)
Definition: test_ir.py:451
nvinfer1::ITensor * maxSequenceSize
Definition: sampleCharRNN.cpp:254
void printShapes(std::ostream &os, const char *phase, const T &shapes)
Definition: sampleOptions.cpp:312
def test_fake_quant_per_channel_other_prec(self)
Definition: quant_conv_transposed_test.py:259
logger
Definition: Polygraphy/polygraphy/logger/logger.py:45
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: reorgPlugin.cpp:200
def test_load_graph(self)
Definition: tf/test_loader.py:35
bool profile
Definition: sampleOptions.h:176
~ParserOnnxConfig()
Definition: parserOnnxConfig.h:66
int mKeepTopK
Definition: generateDetectionPlugin.h:132
constexpr int32_t EnumMax< ActivationType >()
Maximum number of elements in ActivationType enum.
Definition: NvInferRuntimeCommon.h:126
bool teardown()
Used to clean up any state created in the sample class.
Definition: sampleGoogleNet.cpp:198
const char * getPluginType() const override
Return the plugin type.
Definition: lReluPlugin.cpp:110
void log(Severity severity, const char *msg) override
Definition: demo/BERT/infer_c/logging.h:69
def is_empty(self)
Definition: tensor.py:46
int repeat
Definition: sampleUffFasterRCNN.cpp:82
Object used to store and query Tensors after they have been extracted from a Caffe model using the IC...
Definition: NvCaffeParser.h:47
~ResizeNearestPluginCreator()
Definition: resizeNearestPlugin.h:101
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: coordConvACPlugin.cpp:59
const char * getPluginNamespace() const override
Return the namespace of the plugin creator object.
Definition: sampleUffPluginV2Ext.cpp:683
void * copyToDevice(const void *data, size_t count)
Definition: samples/opensource/samplePlugin/fcPlugin.h:243
int getBatchSize() const
Definition: EntropyCalibrator.h:49
Helper for tag dispatching.
Definition: ieee_half.h:300
SampleOnnxMnistCoordConvAC(const samplesCommon::OnnxSampleParams ¶ms)
Definition: sampleOnnxMnistCoordConvAC.cpp:58
int32_t numMoviesPerUser
Definition: sampleMovieLens.cpp:67
bool verifyOutput(const samplesCommon::BufferManager &buffers)
Classifies digits and verify result.
Definition: sampleOnnxMnistCoordConvAC.cpp:279
matmul_k_input_quantizer
Definition: quant_bert.py:74
def test_getitem_out_of_bounds(self, run_results)
Definition: comparator/test_struct.py:53
void serialize(void *buffer) const override
Serialize the layer.
Definition: embLayerNormVarSeqlenPlugin.cpp:377
ComponentWeights::ptr mWeights
Definition: slpProjection.h:51
@ kCONSTANT
Constant layer.
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: nvFasterRCNNPlugin.cpp:268
bool matches(const TRTOption &a, const TRTOption &b)
Matching for TRTOptions is defined as follows:
Definition: getOptions.cpp:47
void setReportableSeverity(Logger::Severity severity)
Definition: logger.cpp:29
float * getBatch()
Definition: frcnnUtils.h:299
bool use_int8_scale_max
Definition: fused_multihead_attention_v2.h:71
PyramidROIAlignPluginCreator()
Definition: pyramidROIAlignPlugin.cpp:35
def __init__(self, path)
Definition: backend/tf/loader.py:139
def tokenize(self, text)
Definition: tokenization.py:226
std::vector< SampleUniquePtr< nvinfer1::IHostMemory > > weightsMemory
Host weights memory holder.
Definition: sampleMNISTAPI.cpp:93
QuantizationFlag
List of valid flags for quantizing the network to int8.
Definition: NvInfer.h:6354
def _symbols_to_sequence(symbols)
Definition: demo/Tacotron2/tacotron2/text/__init__.py:65
p_decoder_dropout
Definition: demo/Tacotron2/tacotron2/model.py:251
nvinfer1::DimsHW argMaxScoreDims
Definition: maskRCNNKernels.h:129
load_engine
Definition: backend/trt_legacy.py:219
def remove(self, x)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:112
def forward(self, input)
Definition: quant_pooling.py:58
float iouThreshold
Definition: maskRCNNKernels.h:87
BatchStream(int batchSize, int maxBatches, std::string prefix, std::vector< std::string > directories)
Definition: BatchStream.h:186
def test_calibrator_iterable_data(self, identity_builder_network)
Definition: test_calibrator.py:39
def parse_batch(self, batch)
Definition: demo/Tacotron2/tacotron2/model.py:621
stft
Definition: waveglow/data_function.py:33
phase
Definition: stft.py:140
constexpr int defaultMaxBatch
Definition: sampleOptions.h:35
static const nvinfer1::Dims3 IMAGE_SHAPE
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:26
T type
Definition: ieee_half.h:381
int fpclassify(half arg)
Classify floating point value.
Definition: ieee_half.h:2720
void skip(int skipCount)
Definition: frcnnUtils.h:281
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: reorgPlugin.cpp:100
def variable_recurrent_factory(inner, reverse=False)
Definition: _functions/quant_rnn.py:149
The SampleINT8API class implements INT8 inference on classification networks.
Definition: sampleINT8API.cpp:76
py::array_t< float > run(py::array_t< int > inputIds, py::array_t< int > segmentIds, py::array_t< int > inputMask)
Definition: infer_c.cpp:36
bool safe
Definition: sampleOptions.h:116
exit_on_errors
Definition: Polygraphy/polygraphy/logger/logger.py:134
A simple imeplementation of the IErrorRecorder interface for use by samples.
Definition: ErrorRecorder.h:37
def load_onnx_weights_and_quant(path, config)
Definition: builder.py:478
@ kNONE
Turn off NVTX traces.
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: proposalLayerPlugin.cpp:343
bool isDebug() const
Definition: sampleConfig.h:306
Single registration point for all plugins in an application. It is used to find plugin implementation...
Definition: NvInferRuntimeCommon.h:958
std::string mNamespace
Definition: groupNormalizationPlugin.h:143
LikelihoodCombinationOperator::ptr mLikelihoodCombinationOperator
Definition: beamSearchPolicy.h:67
half logb(half arg)
Extract exponent.
Definition: ieee_half.h:2676
const void * data() const
Returns pointer to underlying array.
Definition: buffers.h:120
weight_hh
Definition: modules/quant_rnn.py:391
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: proposalLayerPlugin.cpp:192
std::shared_ptr< T > infer_object(T *obj)
Definition: samples/common/common.h:388
virtual nvinfer1::IErrorRecorder * getErrorRecorder() const =0
get the ErrorRecorder assigned to this interface.
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: multilevelProposeROIPlugin.cpp:206
size_t mRpnStride
Definition: proposalPlugin.h:114
const char * COORDCONV_AC_PLUGIN_NAME
Definition: coordConvACPlugin.cpp:28
Definition: quant_pooling.py:62
batch_first
Definition: modules/quant_rnn.py:50
ReorgPluginCreator()
Definition: reorgPlugin.cpp:182
@ kRECURRENCE
Loop Recurrence layer.
void setTextFileName(const char *textFilename)
Set the File Name of the Network Description as a Text File.
Definition: sampleConfig.h:141
constexpr size_t threadsPerCta128
Definition: embLayerNormVarSeqlenPlugin.cpp:32
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: proposalPlugin.cpp:344
Definition: sampleInference.cpp:175
@ kFAILED_COMPUTATION
An error occurred during execution that caused the data to become corrupted, but execution finished.
static std::vector< PluginField > mPluginAttributes
Definition: instanceNormalizationPlugin.h:121
std::shared_ptr< Projection > ptr
Definition: projection.h:35
An RNN layer in a network definition, version 2.
expr sinh(half arg)
Hyperbolic sine.
Definition: ieee_half.h:2473
ResizeNearest(float scale)
Definition: resizeNearestPlugin.cpp:82
const char * getPluginVersion() const override
Return the plugin version.
Definition: normalizePlugin.cpp:234
void operator()(T *obj) const
Definition: samples/common/common.h:378
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: batchedNMSPlugin.cpp:385
size_t totalSize
Definition: maskRCNNKernels.h:186
int32_t topKMovies
Definition: sampleMovieLens.cpp:66
virtual ~IPaddingLayer()
Definition: NvInfer.h:3682
Managed CUDA graph.
Definition: sampleDevice.h:165
bool channelShared
Definition: normalizePlugin.h:99
virtual size_t getWorkspaceSize(int32_t maxBatchSize) const =0
Find the workspace size required by the layer.
std::vector< size_t > mMatchesByOrder
Definition: bleuScoreWriter.h:58
def test_errors_do_not_hang(self)
Definition: test_comparator.py:98
_calib_method
Definition: tensor_quant.py:103
bool saveEngine(const ICudaEngine &engine, const std::string &fileName, std::ostream &err)
Save an engine into a file.
Definition: sampleEngines.cpp:662
def RNNTanhCell(input, hidden, w_ih, w_hh, b_ih=None, b_hh=None)
Definition: _functions/quant_rnn.py:41
virtual bool supportsFormat(DataType type, PluginFormat format) const =0
Check format support.
Definition: quant_conv.py:134
def __getitem__(self, index)
Definition: comparator/data_loader.py:98
int32_t getNbErrors() const noexcept final
Return the number of errors.
Definition: ErrorRecorder.h:46
_amaxs
Definition: max.py:41
def __init__(self, kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=True, **kwargs)
Definition: quant_pooling.py:78
Definition: onnx/test_loader.py:41
def _get_best_indexes(logits, n_best_size)
Definition: data_processing.py:211
SampleCharRNNLoop(SampleCharRNNParams params)
Definition: sampleCharRNN.cpp:257
Definition: demo/Tacotron2/tacotron2/model.py:239
Definition: flattenConcat.h:33
def get_mel(self, filename)
Definition: tacotron2/data_function.py:53
GroupNormalizationPluginCreator()
Definition: groupNormalizationPlugin.cpp:251
void * qkv_ptr
Definition: fused_multihead_attention.h:51
void endCapture(TrtCudaStream &stream)
Definition: sampleDevice.h:197
IPluginV2Ext * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: normalizePlugin.cpp:329
conv1
Definition: tools/pytorch-quantization/tests/fixtures/models.py:47
def test_one_tensor(self, verbose)
Definition: calibrator_test.py:198
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: batchedNMSPlugin.cpp:424
std::string mNamespace
Definition: split.h:180
def version(version_str)
Definition: common.py:19
void serialize(void *buffer) const override
Serialize the layer.
Definition: batchedNMSPlugin.cpp:254
_unsigned
Definition: tensor_quant.py:104
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: gridAnchorPlugin.cpp:290
Definition: samples/common/common.h:335
int mInputTokenCount
Definition: benchmarkWriter.h:49
int vocabularyId
Definition: beamSearchPolicy.h:58
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: gridAnchorPlugin.cpp:274
DeviceBuffer deviceBuffer
Definition: buffers.h:225
def test_input_fake_quant(self)
Definition: quant_pooling_test.py:138
std::vector< bool > mValidSamples
Definition: beamSearchPolicy.h:69
bool next() override
Definition: BatchStream.h:217
def test_can_name_runner(self)
Definition: tf/test_runner.py:27
const char * getPluginType() const override
Return the plugin type.
Definition: instanceNormalizationPlugin.cpp:239
LogStreamConsumer LOG_INFO(const Logger &logger)
produces a LogStreamConsumer object that can be used to log messages of severity kINFO
Definition: pluginLogging.h:485
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace of the plugin creator based on the plugin library it belongs to.
Definition: qkvToContextInt8InterleavedPlugin.cpp:353
virtual int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: samples/opensource/samplePlugin/fcPlugin.h:169
void reportLayerTime(const char *layerName, float timeMs) override
Layer time reporting callback.
Definition: sampleReporting.cpp:243
void generate_pyramid_anchors(const nvinfer1::Dims &image_size)
Definition: proposalLayerPlugin.cpp:272
provides encoder/decoder relevant properties of sequences
Definition: sequenceProperties.h:29
std::string mCalibrationTableName
Definition: EntropyCalibrator.h:89
int combineIndexDimensions(int batchSize, const nvinfer1::Dims &d)
Definition: parserUtils.h:126
const char * getPluginVersion() const override
Return the plugin version.
Definition: instanceNormalizationPlugin.cpp:340
Descriptor for three-dimensional data.
Definition: NvInfer.h:164
def test_api_examples(example)
Definition: Polygraphy/tests/test_examples.py:101
int label
Definition: bboxUtils.h:43
void destroy() override
Destroy the plugin object.
Definition: sampleUffPluginV2Ext.cpp:547
T half_cast(U arg)
Cast to or from half-precision floating point number.
Definition: ieee_half.h:2848
static constexpr int32_t kFORMAT_COMBINATION_LIMIT
Limit on number of format combinations accepted.
Definition: NvInferRuntime.h:453
T type
Definition: samples/common/half.h:324
@ kINTERNAL_ERROR
Internal error has occurred. Execution is unrecoverable.
const char * getPluginName() const override
Return the plugin name.
Definition: multilevelProposeROIPlugin.cpp:54
OptProfileSelector
When setting or querying optimization profile parameters (such as shape tensor inputs or dynamic dime...
Definition: NvInferRuntime.h:998
input_size
Definition: modules/quant_rnn.py:46
gate_layer
Definition: export_tacotron2_onnx.py:129
def __str__(self)
Definition: node.py:105
nvinfer1::DimsHW sortClassScoreDims
Definition: maskRCNNKernels.h:100
bool supportsFormatCombination(int pos, const nvinfer1::PluginTensorDesc *inOut, int nbInputs, int nbOutputs) override
Definition: instanceNormalizationPlugin.cpp:231
def __init__(self, input_size, hidden_size, bias=True, nonlinearity="tanh")
Definition: modules/quant_rnn.py:340
cudaEvent_t mEvent
Definition: sampleDevice.h:153
A tensor in a network definition.
Definition: NvInfer.h:502
IPluginV2Ext * deserializePlugin(const char *name, const void *data, size_t length) override
Called during deserialization of plugin layer.
Definition: specialSlicePlugin.cpp:62
float getFailurePercentage() const
Definition: sampleConfig.h:286
nvinfer1::ITensor * recurrentWeights
Definition: sampleCharRNN.cpp:251
default_quant_map
Definition: quant_modules.py:59
def test_del_index(self, field_names)
Definition: test_ir.py:277
SampleUniquePtr< nvinfer1::IExecutionContext > mContext
Definition: sampleMovieLensMPS.cpp:246
WeightsRole
How a layer uses particular Weights.
Definition: NvInferRuntime.h:685
Definition: sampleCharRNN.cpp:96
int mPreNMSTopK
Definition: proposalLayerPlugin.h:98
const std::string mLayerName
Definition: cropAndResizePlugin.h:100
std::string saveEngine
Definition: sampleUffFasterRCNN.cpp:85
StreamType
Definition: sampleInference.cpp:252
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: regionPlugin.cpp:234
BatchedNMSPlugin(NMSParameters param)
Definition: batchedNMSPlugin.cpp:41
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: regionPlugin.cpp:444
virtual int32_t getDLACore() const noexcept=0
Get the DLA core that the engine executes on.
def text_to_sequence(text, cleaner_names)
Definition: demo/Tacotron2/tacotron2/text/__init__.py:15
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: flattenConcat.cpp:188
conv2
Definition: tools/pytorch-quantization/tests/fixtures/models.py:30
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: groupNormalizationPlugin.cpp:241
constexpr int32_t EnumMax< OptProfileSelector >()
Definition: NvInferRuntime.h:1007
Base class for all layer classes in a network definition.
Definition: NvInfer.h:773
def infer_waveglow_trt(waveglow, waveglow_context, mel, measurements, fp16)
Definition: inference_trt.py:258
static int reportFail(const TestAtom &testAtom)
Definition: samples/common/logging.h:353
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: batchTilePlugin.cpp:73
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleUffSSD.cpp:66
def test_input_fake_quant(self)
Definition: quant_pooling_test.py:48
bool getBatch(void *bindings[], const char *names[], int nbBindings) override
Definition: EntropyCalibrator.h:116
bool help
Definition: sampleMovieLens.cpp:524
Definition: NvInferRuntime.h:325
@ kSIGMOID
Sigmoid activation.
old_indent
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:29
def __init__(self, max_workspace_size=None, tf32=None, fp16=None, int8=None, profiles=None, calibrator=None, strict_types=None)
Definition: backend/trt/loader.py:300
void destroy() override
Destroy the plugin object.
Definition: cropAndResizePlugin.cpp:175
int getBufferSize()
Definition: sampleReformatFreeIO.cpp:193
UffInputOrder
The different possible supported input order.
Definition: NvUffParser.h:45
static const int kIMG_CHANNELS
Definition: sampleFasterRCNN.cpp:87
def infer(self, memory, memory_lengths)
Definition: demo/Tacotron2/tacotron2/model.py:504
static const float BBOX_STD_DEV[]
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:37
static expr log(float arg)
Logarithm implementation.
Definition: samples/common/half.h:1799
virtual void setErrorRecorder(IErrorRecorder *recorder) noexcept=0
Set the ErrorRecorder for this interface.
Plugin class for user-implemented layers.
Definition: NvInferRuntimeCommon.h:581
default_quant_desc_input
Definition: _utils.py:33
Definition: test_data_loader.py:28
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: coordConvACPlugin.cpp:88
bool verbose
Definition: sampleOptions.h:171
const char * getPluginVersion() const override
Return the plugin version.
Definition: multilevelCropAndResizePlugin.cpp:52
nvinfer1::DimsHW sortClassSampleIdxDims
Definition: maskRCNNKernels.h:169
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: cropAndResizePlugin.cpp:173
bool verifyOutput(uint32_t *userInputPtr, uint32_t *, uint32_t *topKItemNumberPtr, float *topKItemProbPtr)
Verifies the inference output with ground truth and logs the results.
Definition: sampleMovieLens.cpp:476
virtual nvinfer1::IBuilderConfig * createBuilderConfig()=0
Create a builder configuration object.
def metric_max_over_ground_truths(metric_fn, prediction, ground_truths)
Definition: evaluate-v1.1.py:64
bool constructNetwork(SampleUniquePtr< nvinfer1::IBuilder > &builder, SampleUniquePtr< nvinfer1::INetworkDefinition > &network, SampleUniquePtr< nvinfer1::IBuilderConfig > &config, SampleUniquePtr< nvcaffeparser1::ICaffeParser > &parser)
Parses a Caffe model for SSD and creates a TensorRT network.
Definition: sampleSSD.cpp:171
unsigned char fused_multihead_attention_v2_int8_192_64_kernel_sm75_cubin[]
Definition: fused_multihead_attention_v2_int8_192_64_kernel.sm75.cpp:19
@ kCAFFE_ROUND_UP
Use CAFFE padding, rounding output size up, uses prePadding value.
expr tan(half arg)
Tangent function.
Definition: ieee_half.h:2431
inputs
Definition: node.py:40
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: batchedNMSPlugin.cpp:95
sampling_rate
Definition: layers.py:61
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:229
std::pair< std::string, T > splitNameAndValue(const std::string &s)
Definition: sampleOptions.cpp:145
def __call__(self)
Definition: backend/trt_legacy.py:130
unsigned int fused_multihead_attention_v2_int8_128_64_kernel_cubin_len
Definition: fused_multihead_attention_v2_int8_128_64_kernel.sm72.cpp:20170
float stepH
Definition: NvInferPluginUtils.h:99
def prepare_input_sequence(texts, cpu_run=False)
Definition: Tacotron2/inference.py:146
def test_is_valid_shape_override(case)
Definition: test_misc.py:57
virtual ~IShapeLayer()
Definition: NvInfer.h:4047
void dumpRefittable(nvinfer1::ICudaEngine &engine)
Log refittable layers and weights of a refittable engine.
Definition: sampleEngines.cpp:617
clip
Definition: tensor_quantizer.py:94
constexpr int32_t EnumMax< RNNDirection >()
Maximum number of elements in RNNDirection enum.
Definition: NvInfer.h:2821
def verbosity(self, severity=CRITICAL)
Definition: Polygraphy/polygraphy/logger/logger.py:169
static PluginFieldCollection mFC
Definition: nmsPlugin.h:112
int mW
Definition: sampleUffPluginV2Ext.cpp:175
static expr lgamma(float arg)
Gamma logarithm implementation.
Definition: samples/common/half.h:2030
def generate_calibration_cache(sequence_lengths, workspace_size, config, weights_dict, squad_json, vocab_file, calibrationCacheFile, calib_num)
Definition: builder.py:623
std::mutex mStackLock
Definition: ErrorRecorder.h:125
std::string onnxFileName
Filename of ONNX file of a network.
Definition: argsParser.h:64
def test_basic_forward(self, verbose)
Definition: quant_rnn_test.py:66
def test_weight_fake_quant_per_tensor(self)
Definition: quant_conv_transposed_test.py:69
IPluginV2Ext * deserializePlugin(const char *name, const void *data, size_t length) override
Called during deserialization of plugin layer.
Definition: pyramidROIAlignPlugin.cpp:73
static const std::vector< float > BACKBONE_STRIDES
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:51
void open()
Definition: sampleMovieLensMPS.cpp:104
PluginCreatorRegistry()
Definition: InferPlugin.cpp:140
static expr fma(float x, float y, float z)
Fused multiply-add implementation.
Definition: samples/common/half.h:1748
def find_nodes_by_ops(graphdef, ops)
Definition: tf/util.py:51
def __init__(self, output_size, **kwargs)
Definition: quant_pooling.py:131
virtual void setErrorRecorder(IErrorRecorder *recorder)=0
Set the ErrorRecorder for this interface.
uint8_t buffer[C *H *W]
Definition: samples/common/common.h:656
bool enableGraph
Definition: demo/BERT/infer_c/common.h:35
GenericBuffer & operator=(GenericBuffer &&buf)
Definition: buffers.h:92
std::string labelsFileName
The file namefo the class labels.
Definition: sampleUffSSD.cpp:51
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: lReluPlugin.cpp:103
def init_decoder_inputs(memory, processed_memory, memory_lengths)
Definition: inference_trt.py:68
size_t getWorkspaceSize(int) const override
Definition: proposalPlugin.cpp:183
virtual IPlugin * createPlugin(const char *layerName, const nvinfer1::Weights *weights, int nbWeights) override
Definition: samples/opensource/samplePlugin/fcPlugin.h:335
convolutions
Definition: demo/Tacotron2/tacotron2/model.py:192
Application-implemented interface for profiling.
Definition: NvInferRuntime.h:665
int64_t mMaxBatchSize
Definition: sampleConfig.h:45
int32_t numAspectRatios
Definition: NvInferPluginUtils.h:94
nvinfer1::Dims getDims() const override
Definition: BatchStream.h:93
A RNN layer in a network definition.
Reorg(int stride)
Definition: reorgPlugin.cpp:36
Definition: proposalLayerPlugin.h:35
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleOnnxMnistCoordConvAC.cpp:55
int32_t & h()
Get the height.
Definition: NvInfer.h:124
Definition: coordConvACPlugin.h:32
Definition: perf_varseqlen.py:28
int * cu_seqlens
Definition: fused_multihead_attention_v2.h:65
nvinfer1::TacticSources enabledTactics
Definition: sampleOptions.h:127
Logger(Severity severity=Severity::kWARNING)
Definition: pluginLogging.h:230
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: normalizePlugin.cpp:277
virtual int32_t initialize()=0
Initialize the layer for execution.
def test_fake_quant_per_tensor(self)
Definition: quant_instancenorm_test.py:63
An ILoopOutputLayer is the sole way to get output from a loop.
Definition: NvInfer.h:4571
Definition: backend/trt/runner.py:30
static std::vector< PluginField > mPluginAttributes
Definition: regionPlugin.h:122
Opaque handle that holds logging information for a particular test.
Definition: samples/common/logging.h:266
std::string getInfo() override
get the textual description of the component
Definition: slpAttention.cpp:73
std::vector< OutputArgs > pargsVec
Definition: sampleMovieLensMPS.cpp:185
int mBackgroundLabel
Definition: proposalLayerPlugin.h:97
const char * getPluginVersion() const override
Return the plugin version.
Definition: priorBoxPlugin.cpp:239
int mBeamWidth
Definition: beamSearchPolicy.h:68
uint8_t * buffer
Definition: sampleReformatFreeIO.cpp:191
MinMax Calibrator.
Definition: NvInfer.h:6093
size_t mOutputSize
Definition: bert_infer.h:350
def test_input_fake_quant(self)
Definition: quant_pooling_test.py:249
nvinfer1::DataType mType
Definition: proposalLayerPlugin.h:110
nvinfer1::ILayer * addLSTMLayers(SampleCharRNNBase::SampleUniquePtr< nvinfer1::INetworkDefinition > &network) final
Add inputs to the TensorRT network and configure LSTM layers using network definition API.
Definition: sampleCharRNN.cpp:662
constexpr size_t xmmasM128
Definition: embLayerNormVarSeqlenPlugin.cpp:37
std::string mOutputFilename
Definition: sampleConfig.h:42
PluginFactory runtimePluginFactory
Definition: samplePlugin.cpp:116
Weights copyToDevice(const void *hostData, size_t count)
Definition: gridAnchorPlugin.cpp:254
_weight_quantizers
Definition: _utils.py:84
def fused_add_tanh_sigmoid_multiply(input_a, input_b, n_channels)
Definition: demo/Tacotron2/waveglow/model.py:23
nvinfer1::IPluginV2 * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: embLayerNormVarSeqlenPlugin.cpp:506
def test_fake_quant_quant_per_channel_other_prec(self)
Definition: quant_conv_test.py:482
const std::string mLayerName
Definition: proposalPlugin.h:107
void reset() override
Reset the reader position, the data reader is ready to read the data from th ebeginning again after t...
Definition: limitedSamplesDataReader.cpp:41
Definition: quant_conv_test.py:460
TBatchStream mStream
Definition: EntropyCalibrator.h:87
Definition: bert_infer.h:33
T type
Definition: ieee_half.h:380
def test_weight_fake_quant_per_tensor(self)
Definition: quant_conv_test.py:288
virtual IConvolutionLayer * addConvolutionNd(ITensor &input, int32_t nbOutputMaps, Dims kernelSize, Weights kernelWeights, Weights biasWeights)=0
Add a multi-dimension convolution layer to the network.
int nbCalBatches
Definition: sampleUffFasterRCNN.cpp:80
an object for parsing ONNX models into a TensorRT network definition
Definition: NvOnnxParser.h:119
Measurement times in milliseconds.
Definition: sampleReporting.h:34
@ kMAX
Maximum of the elements.
static const float RPN_NMS_THRESHOLD
Definition: tlt_mrcnn_config.h:80
BufferDesc desc
Definition: sampleReformatFreeIO.cpp:189
void cuSeqlensToPackedMask(const uint32_t S, const uint32_t B, const uint32_t warps_m, const uint32_t warps_n, const uint32_t warps_k, const int *cuSeqlens, uint32_t *inputMaskX, cudaStream_t stream)
const char * mSemName
Definition: sampleMovieLensMPS.cpp:114
void moveNext()
Definition: sampleInference.cpp:382
Definition: backend/onnxrt/loader.py:1
int mSosId
Definition: vocabulary.h:76
Definition: modules/quant_rnn.py:269
np.dtype get_onnx_tensor_dtype(Union[onnx.ValueInfoProto, onnx.TensorProto] onnx_tensor)
Definition: onnx_importer.py:59
int line
Definition: checkMacrosPlugin.h:95
def load_tf_weights(inputbase, config)
Definition: builder_varseqlen.py:332
int roundUp(int m, int n)
Definition: samples/common/common.h:876
GenericBuffer(nvinfer1::DataType type=nvinfer1::DataType::kFLOAT)
Construct an empty buffer.
Definition: buffers.h:58
def test_fake_quant_input(self)
Definition: quant_conv_transposed_test.py:109
nvinfer1::DimsHW preRefineBboxDims
Definition: maskRCNNKernels.h:163
int find_log_2(int x, bool round_up=false)
Definition: reducedMathPlugin.cpp:39
Definition: sampleMovieLens.cpp:522
InputDataFormat mInputDataFormat
Definition: sampleConfig.h:56
std::string validateTRTOptions(const std::vector< TRTOption > &options)
validateTRTOptions will return a string containing an error message if any options contain non-numeri...
Definition: getOptions.cpp:116
use_fp16
Definition: builder_varseqlen.py:98
def sequence_to_text(sequence)
Definition: demo/Tacotron2/tacotron2/text/__init__.py:43
stream
Definition: backend/trt/runner.py:101
T * mBuffer
Definition: deviceBuffer.h:57
def test_set_severity(self, sev)
Definition: onnxrt/test_runner.py:26
void dumpBuffer(std::ostream &os, const std::string &tensorName)
Dump host buffer with specified tensorName to ostream.
Definition: buffers.h:325
static int reportPass(const TestAtom &testAtom)
Definition: pluginLogging.h:368
def test_fake_quant_input(self)
Definition: quant_conv_transposed_test.py:395
Definition: specialSlicePlugin.h:34
def Calibrator(data_loader, cache=None, BaseClass=trt.IInt8MinMaxCalibrator, batch_size=None)
Definition: tools/Polygraphy/polygraphy/backend/trt/calibrator.py:26
logger
Definition: Polygraphy/polygraphy/logger/logger.py:60
SLPProjection(ComponentWeights::ptr weights)
Definition: slpProjection.cpp:25
const char * getPluginVersion() const override
Return the plugin version.
Definition: nmsPlugin.cpp:287
nvinfer1::DataType getModelDtype() const
Get the Model Data Type.
Definition: sampleConfig.h:89
const char * getPluginType() const override
Return the plugin type.
Definition: pyramidROIAlignPlugin.cpp:117
unsigned int fused_multihead_attention_v2_int8_256_64_kernel_sm75_cubin_len
Definition: fused_multihead_attention_v2_int8_256_64_kernel.sm75.cpp:13968
n_flows
Definition: demo/Tacotron2/waveglow/model.py:165
void setTensorNames()
Definition: blobNameToTensor.h:51
def skipln(prefix, config, init_dict, network, input_tensor, skip)
Definition: builder_varseqlen.py:183
Replicates the algorithm selection using a cache file.
Definition: sampleAlgorithmSelector.cpp:120
owns_context
Definition: backend/trt/runner.py:81
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: nmsPlugin.cpp:96
static expr fmod(float x, float y)
Modulo implementation.
Definition: samples/common/half.h:1648
static PluginFieldCollection mFC
Definition: groupNormalizationPlugin.h:141
~CaffeBufferShutter()
Definition: sampleEngines.cpp:45
virtual const void * getData()=0
int initialize() override
Initialize the layer for execution.
Definition: priorBoxPlugin.cpp:163
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: multilevelProposeROIPlugin.cpp:199
LogStream()
Definition: checkMacrosPlugin.h:58
def test_input_variable_bits(self)
Definition: quant_pooling_test.py:170
def test_basic(self)
Definition: onnx/test_runner.py:28
def init_learn_amax(self)
Definition: tensor_quantizer.py:245
Definition: batchedNMSPlugin.h:32
inputs
Definition: onnx_models.py:37
def get_regression_cutoff(self)
Definition: demo/BERT/helpers/calibrator.py:104
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleUffMNIST.cpp:268
std::unique_ptr< T, TrtDestroyer< T > > TrtUniquePtr
Definition: sampleUtils.h:466
def hasTimestamp(metrics)
Definition: tracer.py:64
@ kUNSPECIFIED_ERROR
An error that does not fall into any other category.
const char * getPluginVersion() const override
Return the plugin version.
Definition: cropAndResizePlugin.cpp:103
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: multilevelProposeROIPlugin.cpp:138
shape
Definition: tensor.py:87
void print(std::ostream &os, void *buf, size_t bufSize, size_t rowCount)
Templated print function that dumps buffers of arbitrary type to std::ostream.
Definition: buffers.h:369
void check_valid_inputs(const nvinfer1::Dims *inputs, int nbInputDims)
Definition: multilevelCropAndResizePlugin.cpp:155
def __init__(self, kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=True, divisor_override=None, **kwargs)
Definition: quant_pooling.py:105
const void * getData() override
Definition: binaryProtoBlob.h:46
nvinfer1::ILogger & getTRTLogger()
Forward-compatible method for retrieving the nvinfer::ILogger associated with this Logger.
Definition: pluginLogging.h:254
def get_shape(tensor)
Definition: onnx/util.py:117
static half_float::half min()
Smallest positive normal value.
Definition: samples/common/half.h:4216
constexpr int32_t EnumMax< TripLimit >()
Maximum number of elements in TripLimit enum.
Definition: NvInfer.h:4516
constexpr size_t threadsPerCta256
Definition: embLayerNormVarSeqlenPlugin.cpp:33
Definition: tools/Polygraphy/polygraphy/logger/__init__.py:1
virtual IReduceLayer * addReduce(ITensor &input, ReduceOperation operation, uint32_t reduceAxes, bool keepDimensions)=0
Add a reduce layer to the network.
def __init__(self, path, explicit_precision=None)
Definition: backend/trt/loader.py:141
virtual void setVerbosityLevel(nvonnxparser::IOnnxConfig::Verbosity verbosity)
Set to specific verbosity Level.
Definition: parserOnnxConfig.h:107
def calibrate_model(model, model_name, data_loader, num_calib_batch, calibrator, hist_percentile, out_dir)
Definition: classification_flow.py:333
std::string mNamespace
Definition: qkvToContextInt8InterleavedPlugin.h:125
def test_clip_gradient(self)
Definition: tensor_quant_test.py:364
def engine_info(engine_filepath)
Definition: trt_utils.py:52
static const char * CROP_AND_RESIZE_PLUGIN_VERSION
Definition: cropAndResizePlugin.cpp:31
def test_volume(case)
Definition: test_misc.py:29
EntropyCalibratorImpl< TBatchStream > mImpl
Definition: EntropyCalibrator.h:132
cublasHandle_t mCublas
Definition: samples/opensource/samplePlugin/fcPlugin.h:316
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: detectionLayerPlugin.cpp:175
The SampleINT8 class implements the INT8 sample.
Definition: sampleINT8.cpp:58
def _get_node_id(self, node)
Definition: graph.py:174
bool supportsFormatCombination(int pos, const nvinfer1::PluginTensorDesc *inOut, int nbInputs, int nbOutputs) override
Definition: groupNormalizationPlugin.cpp:173
IPluginV2Ext * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: flattenConcat.cpp:339
Definition: resizeNearestPlugin.h:34
nvinfer1::ILayer * addMLPLayer(nvinfer1::INetworkDefinition *network, nvinfer1::ITensor &inputTensor, int32_t hiddenSize, nvinfer1::Weights wts, nvinfer1::Weights bias, nvinfer1::ActivationType actType, int idx)
Add an MLP layer.
Definition: sampleMLP.cpp:490
float mIOUThreshold
Definition: generateDetectionPlugin.h:134
padding_mode
Definition: quant_conv.py:208
GroupNormalizationPlugin()=delete
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: batchedNMSPlugin.cpp:506
Definition: tensor_quant.py:236
def test_simple_run_no_fake(self)
Definition: tensor_quantizer_test.py:47
void dumpBindings(const nvinfer1::IExecutionContext &context, std::ostream &os) const
Definition: sampleUtils.h:395
int mHiddenSize
Definition: qkvToContextInt8InterleavedPlugin.h:85
const void * readCalibrationCache(size_t &length)
Definition: EntropyCalibrator.h:66
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: instanceNormalizationPlugin.cpp:160
def __init__(self, file_or_path, keep_ambiguous=True)
Definition: cmudict.py:36
def __init__(self, in_features, out_features, bias=True, **kwargs)
Definition: quant_linear.py:62
size_t getWorkspaceSize(const nvinfer1::PluginTensorDesc *, int, const nvinfer1::PluginTensorDesc *, int) const TRTNOEXCEPT override
Definition: split.h:114
static size_t serialized_size(const char *value)
Definition: serialize.hpp:68
std::vector< int > mMetaData
Definition: componentWeights.h:41
Definition: backend/pyt/loader.py:1
Definition: detectionLayerPlugin.h:108
old
Definition: test_cuda.py:27
UffPoolPluginV2(const void *data, size_t length)
Definition: sampleUffPluginV2Ext.cpp:354
std::vector< int > nonMaximumSuppression(std::vector< std::pair< float, int >> &scoreIndex, float *bbox, const int classNum, const int numClasses, const float nmsThreshold)
Performs non maximum suppression on final bounding boxes.
Definition: sampleFasterRCNN.cpp:419
bool teardown()
Cleans up any state created in the sample class.
Definition: sampleFasterRCNN.cpp:247
virtual ~IMatrixMultiplyLayer()
Definition: NvInfer.h:4224
def test_weight_fake_quant_per_tensor(self)
Definition: quant_conv_test.py:71
void destroy() override
Destroy this ICaffeParser object.
Definition: caffeParser.h:58
def mul(self, a, b)
Definition: 07_creating_a_model_with_the_layer_api/generate.py:46
int clz(int x)
Definition: reducedMathPlugin.cpp:25
int iterations
Definition: sampleOptions.h:149
void setPluginNamespace(const char *libNamespace) override
Set the namespace of the plugin creator based on the plugin library it belongs to.
Definition: plugin.h:63
bool ignore_b1opt
Definition: fused_multihead_attention_v2.h:69
half scalbln(half arg, long exp)
Multiply by power of two.
Definition: ieee_half.h:2659
virtual IPluginCreator * getPluginCreator(const char *pluginType, const char *pluginVersion, const char *pluginNamespace="") noexcept=0
Return plugin creator based on plugin type, version and namespace associated with plugin during netwo...
def disable(self)
Definition: tensor_quantizer.py:150
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: coordConvACPlugin.cpp:69
expr expm1(half arg)
Exponential minus one.
Definition: ieee_half.h:2333
int mConcatAxisID
Definition: flattenConcat.h:102
def test_amax(self)
Definition: tensor_quant_test.py:310
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleMNIST.cpp:51
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleMLP.cpp:60
C
Definition: 07_creating_a_model_with_the_layer_api/generate.py:95
int calculateScore(const samplesCommon::BufferManager &buffers, float *labels, int batchSize, int outputSize, int threshold)
Scores model.
Definition: sampleINT8.cpp:380
def parse_args(parser)
Definition: preprocess_audio2mel.py:23
void transferOutputToHost(TrtCudaStream &stream)
Definition: sampleUtils.h:350
int h
Definition: sampleUffMaskRCNN.cpp:75
uint16 round_half(uint16 value)
Round half-precision number to nearest integer value.
Definition: ieee_half.h:1041
std::shared_ptr< SequenceProperties > ptr
Definition: sequenceProperties.h:32
A elementwise layer in a network definition.
Definition: NvInfer.h:2644
bool isOK()
Definition: caffeWeightFactory.cpp:184
const char * getPluginVersion() const override
Return the plugin version.
Definition: groupNormalizationPlugin.cpp:265
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: sampleUffPluginV2Ext.cpp:419
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: resizeNearestPlugin.cpp:234
def __init__(self, logger, indent)
Definition: Polygraphy/polygraphy/logger/logger.py:44
Definition: buffers.h:188
EntropyCalibratorImpl(BatchStream &stream, int firstBatch, const std::string &networkName, const char *inputBlobName, bool readCache=true)
Definition: frcnnUtils.h:434
Definition: trt/test_loader.py:72
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: normalizePlugin.cpp:190
const char * getPluginType() const override
Return the plugin type.
Definition: detectionLayerPlugin.cpp:148
int mS
Definition: qkvToContextInt8InterleavedPlugin.h:81
bool build()
Function builds the network engine.
Definition: sampleOnnxMnistCoordConvAC.cpp:109
SampleOnnxMNIST(const samplesCommon::OnnxSampleParams ¶ms)
Definition: sampleOnnxMNIST.cpp:52
void fillBuffer(void *buffer, int volume, T min, T max)
Definition: sampleUtils.h:165
use_int8
Definition: builder.py:98
Collect per-layer profile information, assuming times are reported in the same order.
Definition: sampleReporting.h:168
nvinfer1::ILayer * addLSTMCell(SampleUniquePtr< nvinfer1::INetworkDefinition > &network, const LstmIO &inputTensors, nvinfer1::ITensor *sequenceSize, const LstmParams ¶ms, LstmIO &outputTensors)
Definition: sampleCharRNN.cpp:485
GenerateDetectionPluginCreator()
Definition: generateDetectionPlugin.cpp:38
def __init__(self, passed, max_absdiff, max_reldiff)
Definition: compare.py:29
MNISTBatchStream(int batchSize, int maxBatches, const std::string &dataFile, const std::string &labelsFile, const std::vector< std::string > &directories)
Definition: BatchStream.h:43
BuildOptions build
Definition: sampleOptions.h:189
def print_single_query(eval_time_elapsed, prediction, nbest_json)
Definition: inference_varseqlen.py:209
inputs
Definition: backend/cntk/runner.py:34
int dataWidth
Definition: sampleReformatFreeIO.cpp:185
std::string networkTensorsFileName
Definition: sampleINT8API.cpp:655
static const int MAX_PRE_NMS_RESULTS
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:76
CaffeWeightFactory(const trtcaffe::NetParameter &msg, nvinfer1::DataType dataType, std::vector< void * > &tmpAllocs, bool isInitialized)
Definition: caffeWeightFactory.cpp:59
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: batchedNMSPlugin.cpp:464
float percentile
Definition: sampleOptions.h:173
CUresult cuFuncSetAttribute(CUfunction hfunc, CUfunction_attribute attrib, int value) const
Definition: cudaDriverWrapper.cpp:76
IPluginV2Ext * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: coordConvACPlugin.cpp:202
def test_raise(self)
Definition: quant_pooling_test.py:241
T swapEndianness(const T &value)
Definition: samples/common/common.h:296
def __enter__(self)
Definition: graph.py:48
def test_can_get_dtype(self)
Definition: test_ir.py:130
The SampleUffSSD class implements the SSD sample.
Definition: sampleUffSSD.cpp:63
an object containing information about an error
Definition: NvOnnxParser.h:89
ProposalLayerPluginCreator()
Definition: proposalLayerPlugin.cpp:39
The SampleINT8Params structure groups the additional parameters required by the INT8 sample.
Definition: sampleINT8.cpp:47
unk_token
Definition: tokenization.py:335
@ kCUBLAS_LT
cuBLAS LT tactics
unsigned int fused_multihead_attention_int8_128_64_kernel_sm75_cu_o_len
Definition: fused_multihead_attention_int8_128_64_kernel.sm75.cpp:3590
Definition: samples/common/common.h:141
def finish(self, message, mode=LogMode.EACH)
Definition: Polygraphy/polygraphy/logger/logger.py:274
virtual ~IRaggedSoftMaxLayer()
Definition: NvInfer.h:4244
virtual const IBlobNameToTensor * parseBuffers(const char *deployBuffer, std::size_t deployLength, const char *modelBuffer, std::size_t modelLength, nvinfer1::INetworkDefinition &network, nvinfer1::DataType weightType)=0
Parse a deploy prototxt a binaryproto Caffe model from memory buffers to extract network definition a...
const char * getPluginName() const override
Return the plugin name.
Definition: lReluPlugin.cpp:140
def test_range_min_max_equal(self)
Definition: test_data_loader.py:42
def deactivate_impl(self)
Definition: backend/onnx/runner.py:46
static bool isequal(half x, half y)
Comparison implementation.
Definition: samples/common/half.h:2388
def str_from_engine(engine)
Definition: trt/util.py:389
Clock clock_type
Definition: samples/common/common.h:843
int enqueue(const PluginTensorDesc *inputDesc, const PluginTensorDesc *outputDesc, const void *const *inputs, void *const *outputs, void *workspace, cudaStream_t stream) TRTNOEXCEPT override
Execute the layer.
@ kFLOAT
32-bit floating point format.
conv
Definition: print_test.py:47
reduced_divisor()
Definition: reducedMathPlugin.h:57
def __init__(self, sess, timeline_dir=None, name=None)
Definition: backend/tf/runner.py:34
def test_can_convert_in_place_to_variable(self)
Definition: test_ir.py:42
void run(int profIdx, int batchSize, const void *inputIds, const void *segmentIds, const void *inputMask, int warmUps, int iterations)
Definition: bert_infer.h:297
max_absdiff
Definition: compare.py:39
static std::vector< PluginField > mPluginAttributes
Definition: reorgPlugin.h:112
def check_outputs_not_found(not_found, available_outputs)
Definition: trt/util.py:190
static half cast(U arg)
Definition: samples/common/half.h:2622
std::string mName
Definition: samples/common/logging.h:282
hidden_size
Definition: builder.py:93
buf
Definition: infer.py:34
DimsHW(int32_t height, int32_t width)
Construct a DimsHW given height and width.
Definition: NvInfer.h:113
static expr erfc(float arg)
Complementary implementation.
Definition: samples/common/half.h:2018
T * allocMemory(int size=1)
Definition: caffeParser.h:70
Definition: sampleEngines.h:33
def __init__(self, network, config=None)
Definition: backend/trt/loader.py:386
LogStreamConsumerBuffer(std::ostream &stream, const std::string &prefix, bool shouldLog)
Definition: pluginLogging.h:35
std::vector< int > mAnchorsCnt
Definition: multilevelProposeROIPlugin.h:102
tuple shape
Definition: 03_isolating_a_subgraph/generate.py:24
const char * getPluginName() const override
Return the plugin name.
Definition: sampleUffPluginV2Ext.cpp:648
virtual int getBatchSize() const =0
Definition: NvInfer.h:3726
cudnnHandle_t _cudnn_handle
Definition: groupNormalizationPlugin.h:102
def __init__(self, int num_features, float eps=1e-5, float momentum=0.1, bool affine=False, bool track_running_stats=False, **kwargs)
Definition: quant_instancenorm.py:36
int getStartSequenceId() override
Definition: vocabulary.cpp:93
Definition: skipLayerNormInt8InterleavedPlugin.h:94
unsigned char fused_multihead_attention_v2_fp16_128_64_kernel_sm80_cubin[]
Definition: fused_multihead_attention_v2_fp16_128_64_kernel.sm80.cpp:19
nvinfer1::DimsHW sortClassPosDims
Definition: maskRCNNKernels.h:136
static int fpclassify(half arg)
Classification implementation.
Definition: samples/common/half.h:2330
int mBatchCount
The batch that will be read on the next invocation of next()
Definition: BatchStream.h:147
def getTRTType(tensor)
Definition: dumpTFWts.py:54
def __init__(self, vocab_file, do_lower_case=True)
Definition: tokenization.py:187
float * getBatch() override
Definition: BatchStream.h:259
Definition: comparator/data_loader.py:47
def test_import_graph(self, model)
Definition: test_importers.py:93
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: multilevelProposeROIPlugin.cpp:308
static half_float::half max()
Largest finite value.
Definition: samples/common/half.h:4228
Definition: backend/base/loader.py:17
_quant_entry
Definition: quant_modules.py:23
virtual void setAlpha(float alpha)=0
Set the alpha parameter (must be finite).
IPluginV2DynamicExt * createPlugin(const char *name, const nvinfer1::PluginFieldCollection *fc) override
Return a plugin object.
Definition: instanceNormalizationPlugin.cpp:350
void setPluginFactory(nvcaffeparser1::IPluginFactory *factory) override
Set the IPluginFactory used to create the user defined plugins.
Definition: caffeParser.h:48
def __init__(self, name, infer=True)
Definition: onnx-graphsurgeon/tests/test_examples.py:34
def test_raise(self)
Definition: quant_pooling_test.py:220
virtual ~IPluginV2Layer()
Definition: NvInfer.h:3468
def fake_quant(self)
Definition: tensor_quant.py:116
int y
Definition: reducedMathPlugin.h:98
def verbose(self, message, mode=LogMode.EACH)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:223
bias
Definition: modules/quant_rnn.py:430
LogStream< ILogger::Severity::kVERBOSE > gLogVerbose
Definition: checkMacrosPlugin.cpp:51
The SampleINT8APIParams structure groups the additional parameters required by the INT8 API sample.
Definition: sampleINT8API.cpp:55
Definition: quant_instancenorm_test.py:97
@ kCONVOLUTION
Convolution layer.
TrtCudaStream & operator=(const TrtCudaStream &)=delete
uint32_t BuilderFlags
Represents a collection of one or more QuantizationFlag values using binary OR operations,...
Definition: NvInfer.h:6375
def send_on_queue(queue, obj)
Definition: Polygraphy/polygraphy/util/misc.py:211
FusedMultiHeadAttentionXMMAKernelV2(const FusedMultiHeadAttentionKernelMetaInfoV2 *pMetaStart, unsigned int nMetaCount, Data_type type, unsigned int sm)
Definition: fused_multihead_attention_v2.h:426
def __call__(self)
Definition: backend/trt/loader.py:208
def __init__(self, in_channels, out_channels, kernel_size, stride, padding, dilation, transposed, output_padding, groups, bias, padding_mode, quant_desc_input, quant_desc_weight)
Definition: quant_conv.py:245
inv_vocab
Definition: tokenization.py:165
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: pyramidROIAlignPlugin.cpp:98
Plugin factory used to configure plugins.
Definition: NvCaffeParser.h:137
@ kLESS
1 if first operand is less than second operand, 0 otherwise.
def test_loader(self)
Definition: trt/test_loader.py:106
@ kFAILED
The test failed.
@ kPROD
Product of the two elements.
std::string mNamespace
Definition: cropAndResizePlugin.h:125
samplesCommon::CaffeSampleParams mParams
The parameters for the sample.
Definition: sampleMNIST.cpp:96
def get_batch_to_gpu(model_name)
Definition: data_functions.py:49
constexpr int32_t EnumMax< LoopOutput >()
Maximum number of elements in LoopOutput enum.
Definition: NvInfer.h:4501
const char * MULTILEVELCROPANDRESIZE_PLUGIN_NAME
Definition: multilevelCropAndResizePlugin.cpp:32
def __getitem__(self, iteration)
Definition: comparator/data_loader.py:205
PriorBoxPluginCreator()
Definition: priorBoxPlugin.cpp:345
Definition: tools/Polygraphy/polygraphy/backend/tf/__init__.py:1
virtual void run(Fused_multihead_attention_params_v2 ¶ms, cudaStream_t ss) const
Definition: fused_multihead_attention_v2.h:444
def __str__(self)
Definition: histogram.py:241
static std::ostream & severityOstream(Severity severity)
returns an appropriate output stream (cout or cerr) to use with the given severity
Definition: pluginLogging.h:431
std::stack< std::unique_ptr< IPluginCreator > > mRegistry
Definition: InferPlugin.cpp:143
static half_float::half epsilon()
Difference between one and next representable value.
Definition: samples/common/half.h:4234
void visualize_boxes(int img_num, int class_num, std::vector< float > &pred_boxes, std::vector< float > &pred_probs, std::vector< int > &pred_cls_ids, std::vector< int > &box_num_per_img, std::vector< vPPM > &ppms)
Helper function to dump bbox-overlayed images as PPM files.
Definition: sampleUffFasterRCNN.cpp:687
outputs
Definition: backend/tf/loader.py:287
int batch
Definition: sampleOptions.h:148
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: cropAndResizePlugin.cpp:207
nvinfer1::DimsHW sortNMSMarkDims
Definition: maskRCNNKernels.h:105
~DataReader() override=default
def load_tf_weights(inputbase, config)
Definition: builder.py:358
void throwCudnnError(const char *file, const char *function, int line, int status, const char *msg)
Definition: checkMacrosPlugin.cpp:87
def keys(self)
Definition: comparator/struct.py:69
static std::vector< PluginField > mPluginAttributes
Definition: groupNormalizationPlugin.h:142
HostBuffer hostBuffer
Definition: buffers.h:226
virtual IParametricReLULayer * addParametricReLU(ITensor &input, ITensor &slope) noexcept=0
Add a parametric ReLU layer to the network.
def skipln(prefix, config, init_dict, network, input_tensor, skip, bias=None)
Definition: builder.py:178
def assert_min_mse(a, b, tol=1e-20)
Definition: tools/pytorch-quantization/tests/utils.py:110
Definition: sampleUffMaskRCNN.cpp:62
expr cosh(half arg)
Hyperbolic cosine.
Definition: ieee_half.h:2480
def replace_with_clip(self, inputs, outputs)
Definition: replace.py:26
@ kERF
Gauss error function.
timestamp
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:106
Tensor import_tensor(Union[onnx.ValueInfoProto, onnx.TensorProto] onnx_tensor)
Definition: onnx_importer.py:80
Weights * mDeviceHeights
Definition: gridAnchorPlugin.h:98
static half nexttoward(half from, long double to)
Enumeration implementation.
Definition: samples/common/half.h:2305
int gMaxSamplesToRead
Definition: limitedSamplesDataReader.h:41
static const float RPN_ANCHOR_RATIOS[]
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:67
void constructNetwork(SampleUniquePtr< nvinfer1::IBuilder > &builder, SampleUniquePtr< nvinfer1::INetworkDefinition > &network, SampleUniquePtr< nvinfer1::IBuilderConfig > &config, SampleUniquePtr< nvuffparser::IUffParser > &parser)
Parses a Uff model for a MLP NCF model, creates a TensorRT network, and builds a TensorRT engine.
Definition: sampleMovieLens.cpp:210
samplesCommon::ManagedBuffer mInput
Host and device buffers for the input.
Definition: sampleDynamicReshape.cpp:88
EngineCapability
Forward declaration of IPluginFactory for use by other interfaces.
Definition: NvInferRuntime.h:49
Dims4(int32_t d0, int32_t d1, int32_t d2, int32_t d3)
Construct a Dims4 from 4 elements.
Definition: NvInfer.h:310
def check_forward_args(self, input, hidden, batch_sizes)
Definition: modules/quant_rnn.py:152
unsigned int fused_multihead_attention_fp16_96_64_kernel_sm75_cu_o_len
Definition: fused_multihead_attention_fp16_96_64_kernel.sm75.cpp:2409
half copysign(half x, half y)
Take sign.
Definition: ieee_half.h:2702
nvinfer1::DimsHW argMaxBboxDims
Definition: maskRCNNKernels.h:98
def inference(features, tokens)
Definition: inference_c.py:131
batch_size
Definition: backend/trt_legacy.py:157
@ kCHW2
Two wide channel vectorized row major format.
tensor
Definition: test_ir.py:53
int outputSize
Definition: sampleCharRNN.cpp:121
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: resizeNearestPlugin.cpp:187
def info(self, message, mode=LogMode.EACH)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:231
Definition: buffers.h:197
DetectionOutput(DetectionOutputParameters param)
Definition: nmsPlugin.cpp:38
def reset(self)
Definition: histogram.py:87
size_t mTranslationLength
Definition: bleuScoreWriter.h:55
def check_outputs_not_found(not_found, all_outputs)
Definition: onnx/util.py:62
bool infer(std::vector< float > &score, int firstScoreBatch, int nbScoreBatches)
Runs the TensorRT inference engine for this sample.
Definition: sampleINT8.cpp:271
node
Definition: test_ir.py:138
float B
Definition: 07_creating_a_model_with_the_layer_api/generate.py:88
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: coordConvACPlugin.cpp:83
LogStream< ILogger::Severity::kWARNING > gLogWarning
Definition: checkMacrosPlugin.cpp:49
_disabled
Definition: tensor_quantizer.py:83
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: instanceNormalizationPlugin.cpp:218
Definition: calibrator_test.py:94
def inference(features, tokens)
Definition: BERT/inference.py:160
virtual size_t getSerializationSize()=0
Find the size of the serialization buffer required.
Definition: nmsPlugin.h:31
batch_size
Definition: demo/BERT/helpers/calibrator.py:38
const char * getPluginType() const override
Return the plugin type.
Definition: sampleUffPluginV2Ext.cpp:537
bert::cuda_unique_ptr< void > mBetaDev
Definition: skipLayerNormInt8InterleavedPlugin.h:75
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleMovieLens.cpp:277
unsigned char fused_multihead_attention_fp16_96_64_kernel_sm80_cu_o[]
Definition: fused_multihead_attention_fp16_96_64_kernel.sm80.cpp:19
virtual void configureWithFormat(const Dims *inputDims, int32_t nbInputs, const Dims *outputDims, int32_t nbOutputs, DataType type, PluginFormat format, int32_t maxBatchSize)=0
Configure the layer.
output_node
Definition: test_ir.py:114
Definition: sampleUffPluginV2Ext.cpp:645
int64_t mMaxNCalibBatch
Definition: sampleConfig.h:48
void setPluginNamespace(const char *libNamespace) override
Set the namespace that this plugin object belongs to.
Definition: batchedNMSPlugin.cpp:419
IPluginV2DynamicExt * clone() const override
Clone the plugin object.
Definition: batchedNMSPlugin.cpp:397
const char * name
Definition: checkMacrosPlugin.h:98
def main()
Definition: tracer.py:101
int mPooledSize
Definition: pyramidROIAlignPlugin.h:123
def parse_decoder_inputs(self, decoder_inputs)
Definition: demo/Tacotron2/tacotron2/model.py:341
num_layers
Definition: modules/quant_rnn.py:48
_fake_quant
Definition: _utils.py:68
std::vector< SampleUniquePtr< nvinfer1::IHostMemory > > weightsMemory
Definition: sampleCharRNN.cpp:186
def deactivate()
Definition: quant_modules.py:138
virtual nvinfer1::IPluginV2 * createPlugin(const char *layerName, const nvinfer1::Weights *weights, int32_t nbWeights, const char *libNamespace="")=0
Creates a plugin.
def infer(self, x, input_lengths)
Definition: demo/Tacotron2/tacotron2/model.py:220
LogStreamConsumerBuffer(LogStreamConsumerBuffer &&other)
Definition: samples/common/logging.h:44
def unmark_outputs(network, outputs)
Definition: trt/util.py:252
Custom BatchStream class for Faster-RCNN because we use variable input dimensions and different image...
Definition: BatchStream.h:154
static const int DETECTION_MAX_INSTANCES
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:40
int dataWidth
Definition: sampleReformatFreeIO.cpp:118
nvinfer1::ILayer * parseSigmoid(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &, BlobNameToTensor &tensors)
Definition: parseSigmoid.cpp:23
unsigned int fused_multihead_attention_int8_128_64_kernel_sm80_cu_o_len
Definition: fused_multihead_attention_int8_128_64_kernel.sm80.cpp:3267
virtual IScaleLayer * addScale(ITensor &input, ScaleMode mode, Weights shift, Weights scale, Weights power)=0
Add a Scale layer to the network.
unsigned int fused_multihead_attention_v2_int8_192_64_kernel_sm75_cubin_len
Definition: fused_multihead_attention_v2_int8_192_64_kernel.sm75.cpp:20325
Definition: onnxrt/test_runner.py:30
def infer_model(path)
Definition: onnx-graphsurgeon/tests/test_examples.py:70
float init() const override
Definition: softmaxLikelihood.cpp:88
constexpr int32_t EnumMax< UnaryOperation >()
Maximum number of elements in UnaryOperation enum.
Definition: NvInfer.h:3505
bool build()
Builds the network engine.
Definition: sampleGoogleNet.cpp:92
def enable(self)
Definition: tensor_quantizer.py:154
nvinfer1::DataType mType
Definition: buffers.h:173
float y1
Definition: sampleUffMaskRCNN.cpp:54
Definition: trt/test_runner.py:35
void operator()(TrtCudaStream &stream) const
Definition: sampleInference.cpp:242
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: multilevelProposeROIPlugin.cpp:221
int mB
Definition: qkvToContextInt8InterleavedPlugin.h:82
def setup(app)
Definition: onnx-graphsurgeon/docs/conf.py:86
def convert_tokens_to_ids(self, tokens)
Definition: tokenization.py:201
def test_weight_fake_per_tensor(self)
Definition: quant_linear_test.py:53
Definition: backend/onnx/loader.py:1
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: flattenConcat.cpp:204
void * mData
Definition: samples/common/common.h:329
bool setDynamicRange(SampleUniquePtr< nvinfer1::INetworkDefinition > &network)
Sets custom dynamic range for network tensors.
Definition: sampleINT8API.cpp:299
unsigned int mul_coeff
Definition: reducedMathPlugin.h:96
def decode(self, decoder_input, in_attention_hidden, in_attention_cell, in_decoder_hidden, in_decoder_cell, in_attention_weights, in_attention_weights_cum, in_attention_context, memory, processed_memory, mask)
Definition: export_tacotron2_onnx.py:132
def combineDescriptions(prolog, features, descriptions)
Definition: prn_utils.py:29
static const std::vector< float > RPN_ANCHOR_SCALES
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:63
Graph import_graph(graph)
Definition: base_importer.py:21
int mAnchorsCnt
Definition: detectionLayerPlugin.h:100
weight_ih
Definition: modules/quant_rnn.py:431
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: resizeNearestPlugin.cpp:218
static bool isunordered(half x, half y)
Comparison implementation.
Definition: samples/common/half.h:2470
float compute
Definition: sampleReporting.h:54
def main()
Definition: perf.py:39
float seconds() const noexcept
Definition: samples/common/common.h:794
constexpr int32_t EnumMax< DimensionOperation >()
Maximum number of elements in DimensionOperation enum.
Definition: NvInferRuntime.h:310
def calib_data()
Definition: Polygraphy/examples/api/04_int8_calibration_in_tensorrt/example.py:36
virtual void configureWithFormat(const Dims *inputDims, int32_t nbInputs, const Dims *outputDims, int32_t nbOutputs, DataType type, PluginFormat format, int32_t maxBatchSize)=0
Configure the layer.
virtual ~IAlgorithmSelector()
Definition: NvInfer.h:6336
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: nvFasterRCNNPlugin.cpp:229
nvinfer1::IPluginV2DynamicExt * createPlugin(const char *, const nvinfer1::PluginFieldCollection *)
Return a plugin object.
Definition: split.h:158
SequenceProperties()=default
nvinfer1::ILayer * parsePermute(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &, BlobNameToTensor &tensors)
Definition: parsePermute.cpp:23
Definition: samples/common/common.h:375
Definition: onnx/test_loader.py:35
int boxesSize
Definition: batchedNMSPlugin.h:68
virtual int32_t getAll(int32_t size, const char **layerNames, WeightsRole *roles)=0
Get description of all weights that could be refit.
def calib_method(self)
Definition: tensor_quant.py:140
void check_valid_inputs(const nvinfer1::Dims *inputs, int nbInputDims)
Definition: generateDetectionPlugin.cpp:234
static expr log10(float arg)
Common logarithm implementation.
Definition: samples/common/half.h:1807
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: nmsPlugin.cpp:81
CUresult(* _cuModuleGetFunction)(CUfunction *, CUmodule, const char *)
Definition: cudaDriverWrapper.h:77
Layer that represents a Matrix Multiplication.
Definition: NvInfer.h:4186
Definition: sampleUffFasterRCNN.cpp:59
static bool isless(half x, half y)
Comparison implementation.
Definition: samples/common/half.h:2432
const char * getPluginType() const override
Return the plugin type.
Definition: groupNormalizationPlugin.cpp:67
InferenceTime operator+=(InferenceTime &a, const InferenceTime &b)
Definition: sampleReporting.h:107
int initialize() override
Initialize the layer for execution.
Definition: resizeNearestPlugin.cpp:114
static const std::string mEosStr
Definition: vocabulary.h:70
bool readTextProto(trtcaffe::NetParameter *net, const char *file)
Definition: readProto.h:64
constexpr int32_t EnumMax< RNNInputMode >()
Maximum number of elements in RNNInputMode enum.
Definition: NvInfer.h:2849
RefineNMSParameters mParam
Definition: proposalLayerPlugin.h:111
static half logb(half arg)
Exponent implementation.
Definition: samples/common/half.h:2257
~ProposalPluginCreator() override
Definition: proposalPlugin.cpp:332
int initialize() override
Initialize the layer for execution.
Definition: groupNormalizationPlugin.cpp:55
def copy(self)
Definition: tensor.py:177
void * mPtr
Definition: sampleDevice.h:270
std::map< int32_t, std::vector< std::pair< int32_t, float > > > userToExpectedItemProbMap
Definition: sampleMovieLensMPS.cpp:183
int mMaxOrder
Definition: bleuScoreWriter.h:56
void prepare(int profIdx, int batchSize)
Definition: bert_infer.h:158
virtual void run(TKernelParam ¶ms, cudaStream_t ss) const
Definition: fused_multihead_attention.h:258
float confidenceThreshold
Definition: NvInferPluginUtils.h:185
void addAttentionKeys(nvinfer1::INetworkDefinition *network, nvinfer1::ITensor *memoryStates, nvinfer1::ITensor **attentionKeys) override
add attention keys calculation (from source memory states) to the network
Definition: multiplicativeAlignment.cpp:48
bool isless(half x, half y)
Comparison for less than.
Definition: ieee_half.h:2795
virtual __attribute__((deprecated)) virtual std IHostMemory * serialize() const noexcept=0
Get the amount of workspace the engine uses.
fc1
Definition: tools/pytorch-quantization/tests/fixtures/models.py:49
nvinfer1::IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: cropAndResizePlugin.cpp:181
def test_strict_types(self, identity_builder_network, flag)
Definition: trt/test_loader.py:193
int W
Definition: regionPlugin.h:97
def __setitem__(self, index, elem)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:91
Definition: sampleCharRNN.cpp:241
def read_calibration_cache(self)
Definition: demo/BERT/helpers/calibrator.py:89
std::shared_ptr< LikelihoodCombinationOperator > ptr
Definition: likelihoodCombinationOperator.h:27
padding_mode
Definition: quant_conv.py:164
static expr sin(float arg)
Sine implementation.
Definition: samples/common/half.h:1886
dtype
Definition: tensor.py:86
expr tanh(half arg)
Hyperbolic tangent.
Definition: ieee_half.h:2487
static T cast_impl(float arg, true_type)
Definition: samples/common/half.h:2672
int mKeepTopK
Definition: detectionLayerPlugin.h:128
exclude_outputs
Definition: backend/onnx/loader.py:126
def test_import_graph_tensor_map_preserved(self)
Definition: test_importers.py:105
~BinaryProtoBlob() override
Definition: binaryProtoBlob.h:56
def __init__(self, uff_loader, uff_order=None)
Definition: backend/trt_legacy.py:81
logging_indent
Definition: Polygraphy/polygraphy/logger/logger.py:127
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: gridAnchorPlugin.cpp:322
def parse_args(parser)
Definition: export_tacotron2_onnx.py:29
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: nvFasterRCNNPlugin.cpp:354
Definition: quant_pooling.py:36
virtual DataType type() const noexcept=0
The type of the memory that was allocated.
IPluginV2Ext * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: cropAndResizePlugin.cpp:295
std::vector< std::string > positionalArgs
Definition: getOptions.h:61
int mSM
Definition: qkvToContextInt8InterleavedPlugin.h:83
void shutdownProtobufLibrary(void)
Shuts down protocol buffers library.
bool teardown()
Cleans up any state created in the sample class.
Definition: sampleSSD.cpp:264
virtual bool enqueue(int32_t batchSize, void **bindings, cudaStream_t stream, cudaEvent_t *inputConsumed) noexcept=0
Asynchronously execute inference on a batch.
static expr erf(float arg)
Error function implementation.
Definition: samples/common/half.h:2006
EntropyCalibratorImpl mImpl
Definition: frcnnUtils.h:537
float * aspectRatios
Definition: NvInferPluginUtils.h:93
outputs
Definition: backend/trt_legacy.py:155
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: samples/opensource/samplePlugin/fcPlugin.h:95
def test_empty_tensor_with_dynamic_input_shape_tensor(self)
Definition: trt/test_runner.py:99
def __init__(self, logger, severity)
Definition: Polygraphy/polygraphy/logger/logger.py:59
void loadXMMAKernels()
Definition: fused_multihead_attention.h:212
def test_basic(self)
Definition: test_postprocess.py:21
virtual void destroy()=0
Destroy IOnnxConfig object.
bool mIgnoreBatch
Definition: flattenConcat.h:127
int enqueue(int batch_size, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: resizeNearestPlugin.cpp:197
size_t mCropWidth
Definition: cropAndResizePlugin.h:101
virtual int32_t getNbInputs() const =0
Return number of inputs of the algorithm.
TrtCudaEvent gpuStart
Definition: sampleInference.cpp:170
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: normalizePlugin.cpp:282
def test_import_tensor_unknown_metadata(self)
Definition: test_importers.py:54
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: gridAnchorPlugin.cpp:193
def load_wav_to_torch(full_path)
Definition: demo/Tacotron2/common/utils.py:31
def __add__(self, List[object] other_list)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:129
def __init__(self, str op, str name=None, Dict[str, object] attrs=None, List["Tensor"] inputs=None, List["Tensor"] outputs=None)
Definition: node.py:25
virtual int read(int samplesToRead, int maxInputSequenceLength, int *hInputData, int *hActualInputSequenceLengths)=0
reads the batch of smaples/sequences
int32_t numUsers
Definition: sampleMovieLens.cpp:65
Definition: quant_rnn_test.py:55
def test_device_buffer_resize(self, shapes)
Definition: test_cuda.py:41
nvinfer1::DataType mDataType
Definition: samples/opensource/samplePlugin/fcPlugin.h:311
LogStreamConsumer(LogStreamConsumer &&other)
Definition: pluginLogging.h:142
RPROIParams is used to create the RPROIPlugin instance.
Definition: NvInferPluginUtils.h:116
int read(int samplesToRead, int maxInputSequenceLength, int *hInputData, int *hActualInputSequenceLengths) override
reads the batch of smaples/sequences
Definition: textReader.cpp:32
int32_t topK
Definition: NvInferPluginUtils.h:184
def infer_impl(self, feed_dict)
Definition: backend/trt_legacy.py:334
static PluginFieldCollection mFC
Definition: lReluPlugin.h:94
DimensionType
The type of data encoded across this dimension.
Definition: NvInferRuntimeCommon.h:165
def activate_impl(self)
Definition: backend/pyt/runner.py:43
@ kHIDDEN
Hidden gate (h).
def is_shape_dynamic(shape)
Definition: Polygraphy/polygraphy/util/misc.py:143
static void help(std::ostream &out)
Definition: sampleOptions.cpp:1167
def emb_layernorm(builder, network, config, weights_dict, builder_config, sequence_lengths, batch_sizes)
Definition: builder.py:528
Definition: compare.py:58
PriorBoxParameters mParam
Definition: priorBoxPlugin.h:92
Definition: maskRCNNKernels.h:189
IPluginV2Ext * deserializePlugin(const char *name, const void *data, size_t length) override
Called during deserialization of plugin layer.
Definition: proposalLayerPlugin.cpp:98
half fmax(half x, half y)
Maximum of half expressions.
Definition: ieee_half.h:2289
SampleMLPParams mParams
The parameters for the sample.
Definition: sampleMLP.cpp:85
size_t mInputHeight
Definition: cropAndResizePlugin.h:101
Wrapper for binary half-precision functions needing specialization for individual argument types.
Definition: ieee_half.h:1051
bool writeASCIIFile(const std::string &fileName, const std::vector< T > &in)
Definition: samples/common/common.h:467
_if_quant
Definition: tensor_quantizer.py:84
void setPluginNamespace(const char *libNamespace) override
Set the namespace that this plugin object belongs to.
Definition: multilevelCropAndResizePlugin.cpp:145
def test_per_channel_scale(self)
Definition: tensor_quant_test.py:151
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: pyramidROIAlignPlugin.cpp:266
RefineNMSParameters mParam
Definition: generateDetectionPlugin.h:104
void terminate() override
Release resources acquired during plugin layer initialization.
static bool islessgreater(half x, half y)
Comparison implementation.
Definition: samples/common/half.h:2456
def test_scale_amax(self)
Definition: tensor_quantizer_test.py:117
Definition: reducedMathPlugin.h:54
nvinfer1::IPluginV2DynamicExt * clone() const override
Clone the plugin object.
Definition: qkvToContextInt8InterleavedPlugin.cpp:93
def forward(self, input, output_size=None)
Definition: quant_conv.py:297
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: proposalLayerPlugin.cpp:337
def test_multirun_outputs_are_different(self)
Definition: test_comparator.py:122
virtual ~IAlgorithmContext()
Definition: NvInfer.h:6255
bool verifyOutput(const samplesCommon::BufferManager &buffers)
Classifies digits and verify result.
Definition: sampleMLP.cpp:330
int mFeatureLength
Definition: pyramidROIAlignPlugin.h:96
static PluginFieldCollection mFC
Definition: generateDetectionPlugin.h:130
def enable_clip(self)
Definition: tensor_quantizer.py:163
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: coordConvACPlugin.cpp:150
def forward(self, inputs)
Definition: clip.py:56
void configurePlugin(const nvinfer1::DynamicPluginTensorDesc *in, int nbInputs, const nvinfer1::DynamicPluginTensorDesc *out, int nbOutputs) override
Definition: qkvToContextInt8InterleavedPlugin.cpp:147
cudnnPoolingMode_t mMode
Definition: sampleUffPluginV2Ext.cpp:635
cache_file
Definition: demo/BERT/helpers/calibrator.py:33
decoder_rnn_dim
Definition: demo/Tacotron2/tacotron2/model.py:246
Definition: backend/trt/loader.py:183
DimsExprs getOutputDimensions(int index, const nvinfer1::DimsExprs *inputs, int nbInputDims, nvinfer1::IExprBuilder &exprBuilder) override
Definition: groupNormalizationPlugin.cpp:82
std::unordered_set< std::string > names
Definition: sampleCharRNN.cpp:82
Holds properties for configuring a builder to produce an engine.
Definition: NvInfer.h:6464
std::vector< bool > mActive
Definition: sampleInference.cpp:466
static const int kIMG_W
Definition: sampleFasterRCNN.cpp:89
def get_input_metadata(self)
Definition: backend/onnx/runner.py:62
bool enable_i2f_trick
Definition: fused_multihead_attention.h:85
const char * getPluginVersion() const override
Return the plugin version.
Definition: sampleUffPluginV2Ext.cpp:542
void configurePlugin(const nvinfer1::DynamicPluginTensorDesc *in, int nbInputs, const nvinfer1::DynamicPluginTensorDesc *out, int nbOutputs) override
def validate(run_results, check_finite=None, check_nan=None, fail_fast=None)
Definition: comparator/comparator.py:281
Definition: sampleOptions.h:134
void launch(TrtCudaStream &stream)
Definition: sampleDevice.h:192
ComponentWeights::ptr mWeights
Definition: lstmEncoder.h:50
def get_model_config(model_name, args)
Definition: demo/Tacotron2/models.py:85
matmul_a_input_quantizer
Definition: quant_bert.py:76
void convertAndCopyToBuffer(char *&buffer, const nvinfer1::Weights &weights)
Definition: samples/opensource/samplePlugin/fcPlugin.h:277
MultilevelProposeROI(int prenms_topk, int keep_topk, float fg_threshold, float iou_threshold, const nvinfer1::Dims image_size)
Definition: multilevelProposeROIPlugin.cpp:111
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The pointer to the engine.
Definition: buffers.h:455
artifacts
Definition: Polygraphy/tests/test_examples.py:57
@ kUNARY
UnaryOp operation Layer.
def test_fake_quant_per_channel_bias(self)
Definition: quant_conv_transposed_test.py:286
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: detectionLayerPlugin.cpp:143
def infer_impl(self, feed_dict)
Definition: backend/pyt/runner.py:48
The SampleFasterRCNN class implements the FasterRCNN sample.
Definition: sampleFasterRCNN.cpp:55
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: embLayerNormVarSeqlenPlugin.cpp:407
The SampleFasterRCNNParams structure groups the additional parameters required by the FasterRCNN samp...
Definition: sampleFasterRCNN.cpp:45
Logger(Severity severity=Severity::kWARNING)
Definition: samples/common/logging.h:209
Configuration Manager Class Concrete Implementation.
Definition: parserOnnxConfig.h:40
std::vector< nvinfer1::Weights > mGateKernelWeights
Definition: lstmEncoder.h:51
def start(self, message, mode=LogMode.EACH)
Definition: Polygraphy/polygraphy/logger/logger.py:270
std::string mPluginName
Definition: sampleUffPluginV2Ext.cpp:690
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: generateDetectionPlugin.cpp:296
const char * mPluginNamespace
Definition: coordConvACPlugin.h:88
def test_set_default_quant_desc(self)
Definition: quant_conv_test.py:237
dtype
Definition: isolate.py:54
const binary_t binary
Tag for binary construction.
Definition: ieee_half.h:349
def get_graph_output_names(graph)
Definition: tf/util.py:157
def build_two_layer_graph()
Definition: test_ir.py:347
def __init__(self, engine, name=None)
Definition: backend/trt/runner.py:34
Definition: comparator/struct.py:128
Definition: comparator/data_loader.py:1
void destroy() override
Destroy the plugin object.
Definition: detectionLayerPlugin.cpp:138
void readGeneratedResult(int sampleCount, int maxOutputSequenceLength, int *hOutputData, int *hActualOutputSequenceLengths)
Definition: beamSearchPolicy.cpp:129
Definition: cropAndResizePlugin.h:105
int32_t expectedPredictedMaxRatingItem
Definition: sampleMovieLens.cpp:52
buf
Definition: perf.py:30
nvinfer1::DataType mDataType
Definition: caffeWeightFactory.h:61
Definition: sampleOptions.h:74
RPROIPluginCreator()
Definition: nvFasterRCNNPlugin.cpp:311
def __call__(self)
Definition: backend/tf/loader.py:328
constexpr int32_t EnumMax< TacticSource >()
Maximum number of tactic sources in TacticSource enum.
Definition: NvInfer.h:6446
@ kLESS
Check if element in first tensor is less than corresponding element in second tensor.
std::string readNextLine(std::ifstream &file, char delim)
Helper function to read the next line of the MovieLens dataset .csv file and return the contents of t...
Definition: sampleMovieLens.cpp:361
int mNumber
The number to classify.
Definition: sampleMNISTAPI.cpp:89
def unique_list(sequence)
Definition: Polygraphy/polygraphy/util/misc.py:60
float * aspectRatios
Definition: NvInferPluginUtils.h:146
def is_shape_dynamic(shape)
Definition: trt_utils.py:24
run_options
Definition: backend/tf/runner.py:55
logger_callbacks
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:108
Definition: tools/pytorch-quantization/pytorch_quantization/calib/calibrator.py:1
static expr atanh(float arg)
Hyperbolic area tangent implementation.
Definition: samples/common/half.h:1994
~SoftmaxLikelihoodCombinationOperator() override=default
void dumpOutputs(const nvinfer1::IExecutionContext &context, std::ostream &os) const
Definition: sampleUtils.h:389
IPluginV2DynamicExt * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: groupNormalizationPlugin.cpp:285
bool indexCheck(int32_t index) const noexcept
Definition: ErrorRecorder.h:117
void destroy() override
Destroy the plugin object.
Definition: batchedNMSPlugin.cpp:375
PluginRegistrar()
Definition: NvInferRuntimeCommon.h:1400
def convert_convinv_1d_to_2d(convinv)
Definition: export_waveglow_onnx.py:42
int inputH
The input height.
Definition: sampleMLP.cpp:47
def activate_impl(self)
Definition: backend/base/runner.py:75
def test_unmark_shape_outputs(self, reshape_network)
Definition: trt/test_loader.py:154
def num_dynamic_dimensions(shape)
Definition: Polygraphy/polygraphy/util/misc.py:139
Definition: demo/BERT/helpers/calibrator.py:27
def test_o(self)
Definition: test_ir.py:205
def VariableRecurrentReverse(inner)
Definition: _functions/quant_rnn.py:198
void setReferenceFileName(const char *referenceFilename)
Definition: sampleConfig.h:245
BatchStream(int batchSize, int maxBatches, std::string prefix, std::string suffix, std::vector< std::string > directories)
Definition: BatchStream.h:157
def collect(self, x)
Definition: max.py:50
RefineNMSParameters mParam
Definition: multilevelProposeROIPlugin.h:115
std::string mNamespace
Definition: embLayerNormVarSeqlenPlugin.h:137
CropAndResizePlugin()=delete
Region(RegionParameters params)
Definition: regionPlugin.cpp:79
static std::unordered_map< std::string, LayerParseFn > gParseTable
Definition: opParsers.h:75
virtual const char * getName() const =0
Return the name of a layer.
float x2
Definition: samples/common/common.h:669
unsigned int fused_multihead_attention_v2_fp16_128_64_kernel_sm80_cubin_len
Definition: fused_multihead_attention_v2_fp16_128_64_kernel.sm80.cpp:6078
bool reportError(ErrorCode val, IErrorRecorder::ErrorDesc desc) noexcept final
report an error to the error recorder with the corresponding enum and description.
Definition: ErrorRecorder.h:85
def concat(self, inputs, axis=0)
Definition: 09_shape_operations_with_the_layer_api/generate.py:46
Definition: tools/Polygraphy/polygraphy/backend/base/__init__.py:1
virtual nvinfer1::Weights operator()(const std::string &layerName, WeightType weightType)
Definition: caffeWeightFactory.cpp:148
max_wav_value
Definition: tacotron2/data_function.py:35
std::ostream & printBatch(std::ostream &os, int maxBatch)
Definition: sampleOptions.cpp:327
Definition: quant_pooling_test.py:107
const char * getPluginVersion() const override
Return the plugin version.
Definition: pyramidROIAlignPlugin.cpp:48
void destroy() override
Destroy the plugin object.
Definition: multilevelProposeROIPlugin.cpp:201
std::vector< char > mCalibrationCache
Definition: sampleEngines.cpp:187
const char * getPluginName() const override
Return the plugin name.
Definition: pyramidROIAlignPlugin.cpp:43
void preprocessPPM(PPM< uint8_t > &src, PPM< uint8_t > &dst, int target_h, int target_w)
Definition: sampleUffMaskRCNN.cpp:161
std::map< std::string, nvinfer1::Weights > loadWeights(const std::string file)
Load requested weights from a formatted file into a map.
Definition: sampleCharRNN.cpp:354
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: flattenConcat.cpp:265
def test_cuda_ext_inplace(self)
Definition: tensor_quant_test.py:230
user_input_metadata
Definition: comparator/data_loader.py:88
expr cbrt(half arg)
Cubic root.
Definition: ieee_half.h:2386
cudaGraph_t mGraph
Definition: sampleDevice.h:205
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: regionPlugin.cpp:371
_weight_quantizer
Definition: _utils.py:81
severity
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:45
static int reportWaive(const TestAtom &testAtom)
Definition: pluginLogging.h:380
static void reportTestEnd(const TestAtom &testAtom, TestResult result)
Report that a test has ended.
Definition: pluginLogging.h:361
void operator()(TrtCudaStream &stream) const
Definition: sampleInference.cpp:201
def __iadd__(self, List[object] other_list)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:133
int H
Definition: reorgPlugin.h:87
const char * getPluginType() const override
Return the plugin type.
Definition: batchTilePlugin.cpp:152
static const std::string MODEL_NAME
Definition: tlt_mrcnn_config.h:170
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: multilevelProposeROIPlugin.cpp:449
unsigned int fused_multihead_attention_v2_int8_384_64_kernel_sm80_cubin_len
Definition: fused_multihead_attention_v2_int8_384_64_kernel.sm80.cpp:17244
def enable_calib(self)
Definition: tensor_quantizer.py:176
bool reshapeWeights(const Weights &input, const int32_t *shape, const int32_t *shapeOrder, void *data, int32_t nbDims)
Reformat the input weights of the given shape based on the new order of dimensions.
allow_growth
Definition: backend/tf/loader.py:370
virtual ~IResizeLayer()
Definition: NvInfer.h:4483
void printTiming(const std::vector< InferenceTime > &timings, int runsPerAvg, std::ostream &os)
Print a timing trace.
Definition: sampleReporting.cpp:85
The SampleOnnxMnistCoordConvAC class implements the ONNX MNIST sample.
Definition: sampleOnnxMnistCoordConvAC.cpp:52
float negSlope
Definition: lReluPlugin.h:95
cudnnHandle_t mCudnn
Definition: sampleUffPluginV2Ext.cpp:631
bool checkEraseRepeatedOption(Arguments &arguments, const std::string &option, std::vector< T > &values)
Definition: sampleOptions.cpp:204
Definition: multilevelProposeROIPlugin.h:120
bool mClipBoxes
Definition: batchedNMSPlugin.h:72
static size_t serialized_size(T const &value)
Definition: serialize.hpp:47
constexpr int defaultAvgTiming
Definition: sampleOptions.h:38
bool verifyOutput(const samplesCommon::BufferManager &buffers)
Filters output detections and verify results.
Definition: sampleSSD.cpp:316
Severity mReportableSeverity
Definition: demo/BERT/infer_c/logging.h:75
def totalData(features, profile)
Definition: profiler.py:53
def test_backward(self)
Definition: functional_test.py:69
def lowercase(text)
Definition: cleaners.py:71
int32_t nbFields
Number of PluginField entries.
Definition: NvInferRuntimeCommon.h:878
Plugin factory used to configure plugins.
Definition: NvCaffeParser.h:88
def test_load_pb(self)
Definition: tf/test_loader.py:45
def write_calibration_cache(self, cache)
Definition: demo/BERT/helpers/calibrator.py:95
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: gridAnchorPlugin.cpp:191
static const std::vector< std::string > MODEL_OUTPUTS
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:173
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: gridAnchorPlugin.cpp:344
def reset(self)
Definition: max.py:84
Definition: calibrator_test.py:196
def run_results()
Definition: comparator/test_struct.py:14
@ kMODEL_DESERIALIZE_FAILED
virtual void setPluginFactoryV2(IPluginFactoryV2 *factory)=0
Set the IPluginFactoryV2 used to create the user defined pluginV2 objects.
def validate(model, criterion, valset, epoch, batch_iter, batch_size, world_size, collate_fn, distributed_run, rank, batch_to_gpu)
Definition: train.py:260
SampleCharRNNBase(const SampleCharRNNParams ¶ms)
Definition: sampleCharRNN.cpp:148
def VariableRecurrent(inner)
Definition: _functions/quant_rnn.py:156
nvinfer1::DimsHW sortNMSMarkDims
Definition: maskRCNNKernels.h:137
std::string getInfo() override
get the textual description of the component
Definition: bleuScoreWriter.cpp:224
stream
Definition: backend/trt_legacy.py:312
@ DATA_TYPE_FP16
Definition: fused_multihead_attention_common.h:27
std::string outputProposalName
Definition: sampleUffFasterRCNN.cpp:65
CoordConvACPlugin()
Definition: coordConvACPlugin.cpp:34
def test_full_range(self)
Definition: tensor_quant_test.py:259
Definition: cudaDriverWrapper.h:30
def test_in_feature_fake_quant(self)
Definition: quant_conv_test.py:112
static expr acos(float arg)
Arc cosine implementation.
Definition: samples/common/half.h:1918
def shapes_match(shape0, shape1)
Definition: Polygraphy/polygraphy/util/misc.py:157
T type
Definition: ieee_half.h:386
def test_resnet50(self, request, pytestconfig)
Definition: classification_flow_test.py:29
void insertShapesBuild(std::unordered_map< std::string, ShapeRange > &shapes, nvinfer1::OptProfileSelector selector, const std::string &name, const std::vector< int > &dims)
Definition: sampleOptions.cpp:217
def expand_abbreviations(text)
Definition: cleaners.py:61
const std::string RNNW_L1_NAME
Definition: sampleCharRNN.cpp:75
def __init__(self, mask_padding, n_mel_channels, n_symbols, symbols_embedding_dim, encoder_kernel_size, encoder_n_convolutions, encoder_embedding_dim, attention_rnn_dim, attention_dim, attention_location_n_filters, attention_location_kernel_size, n_frames_per_step, decoder_rnn_dim, prenet_dim, max_decoder_steps, gate_threshold, p_attention_dropout, p_decoder_dropout, postnet_embedding_dim, postnet_kernel_size, postnet_n_convolutions, decoder_no_early_stopping)
Definition: demo/Tacotron2/tacotron2/model.py:588
static std::string generateText(int sequenceLength, const int *currentOutputData, Vocabulary::ptr vocabulary)
Definition: dataWriter.cpp:23
SkipLayerNormInterleavedPlugin()=delete
virtual const IDimensionExpr * operation(DimensionOperation op, const IDimensionExpr &first, const IDimensionExpr &second)=0
Return pointer to IDimensionExp that represents the given operation applied to first and second.
Describes the context and requirements, that could be fulfilled by one or more instances of IAlgorith...
Definition: NvInfer.h:6227
bool build()
Function builds the network engine.
Definition: sampleOnnxMNIST.cpp:103
Helper class for half casts.
Definition: ieee_half.h:1052
input_metadata
Definition: comparator/data_loader.py:274
dropout
Definition: modules/quant_rnn.py:51
const char * getPluginType() const override
Return the plugin type.
Definition: coordConvACPlugin.cpp:129
def __init__(self, *args, **kwargs)
Definition: modules/quant_rnn.py:274
const int mSeqLength
Definition: bert_infer.h:337
Class which manages logging of TensorRT tools and samples.
Definition: demo/BERT/infer_c/logging.h:28
virtual ~IAlgorithmIOInfo()
Definition: NvInfer.h:6188
_network
Definition: backend/trt/loader.py:406
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: batchTilePlugin.cpp:178
def test_clip_gradient(self)
Definition: tensor_quant_test.py:115
def bert_model(config, init_dict, network, input_tensor, input_mask)
Definition: builder.py:323
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: generateDetectionPlugin.cpp:61
float spatialScale
Definition: NvInferPluginUtils.h:127
~PyramidROIAlignPluginCreator()
Definition: pyramidROIAlignPlugin.h:109
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: proposalLayerPlugin.cpp:255
def _remove_from_elem(self, elem)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:81
std::string uffFileName
The file name of the UFF model to use.
Definition: sampleUffSSD.cpp:50
void serialize(void *buffer) const override
Serialize the layer.
Definition: normalizePlugin.cpp:123
virtual size_t getWorkspaceSize(const PluginTensorDesc *inputs, int32_t nbInputs, const PluginTensorDesc *outputs, int32_t nbOutputs) const =0
Find the workspace size required by the layer.
static int reportTest(const TestAtom &testAtom, bool pass)
Definition: pluginLogging.h:386
std::string calibrationBatches
The path to calibration batches.
Definition: sampleSSD.cpp:53
Definition: demo/Tacotron2/common/utils.py:1
Definition: demo/Tacotron2/waveglow/model.py:32
unsigned int fused_multihead_attention_v2_fp16_64_64_kernel_sm80_cubin_len
Definition: fused_multihead_attention_v2_fp16_64_64_kernel.sm80.cpp:1653
Definition: backend/onnx/runner.py:25
path
Definition: Polygraphy/tests/test_examples.py:56
virtual ~IExecutionContext() noexcept
Definition: NvInferRuntime.h:1660
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: normalizePlugin.cpp:100
virtual TensorFormat getTensorFormat() const =0
Return TensorFormat of the input/output of algorithm.
def deactivate_impl(self)
Definition: backend/onnxrt/runner.py:43
static void help(std::ostream &out)
Definition: sampleOptions.cpp:1241
std::vector< void * > & mTmpAllocs
Definition: caffeWeightFactory.h:60
value
Definition: quant_bert.py:66
Definition: tools/Polygraphy/polygraphy/backend/trt/calibrator.py:1
static expr fabs(float arg)
Definition: samples/common/half.h:2536
def test_can_cast_dtype(self)
Definition: test_data_loader.py:99
def _get_runner_pair(self, runner_pair)
Definition: comparator/struct.py:169
def compute_amax(model, **kwargs)
Definition: classification_flow.py:400
static const int FPN_CLASSIF_FC_LAYERS_SIZE
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:54
static std::vector< PluginField > mPluginAttributes
Definition: proposalLayerPlugin.h:138
static std::ostream & severityOstream(Severity severity)
returns an appropriate output stream (cout or cerr) to use with the given severity
Definition: samples/common/logging.h:410
bool build(IAlgorithmSelector *selector)
Builds the network engine.
Definition: sampleAlgorithmSelector.cpp:375
float maxSize
Definition: NvInferPluginUtils.h:145
static expr cos(float arg)
Cosine implementation.
Definition: samples/common/half.h:1894
void write(char *&buffer, const T &val)
Definition: samples/opensource/samplePlugin/fcPlugin.h:230
@ kGRU
Three-gate network consisting of Gated Recurrent Units.
def print_single_query(eval_time_elapsed, prediction, nbest_json)
Definition: inference_c.py:182
convinv
Definition: demo/Tacotron2/waveglow/model.py:170
def test_grow(self, verbose)
Definition: calibrator_test.py:96
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: nmsPlugin.cpp:103
float smallerThanMinimalLikelihood() const override
Definition: softmaxLikelihood.cpp:93
Definition: Polygraphy/polygraphy/common/exception.py:1
bool teardown()
Cleans up any state created in the sample class.
Definition: sampleINT8.cpp:357
int * mNumPriors
Definition: gridAnchorPlugin.h:97
nvinfer1::Dims mInputDims
The dimensions of the input to the network.
Definition: samplePlugin.cpp:114
virtual ~IAlgorithmVariant()
Definition: NvInfer.h:6216
def main()
Definition: onnx-graphsurgeon/setup.py:33
def test_non_matching_outputs(self)
Definition: test_compare.py:86
int getSourceStatesSize() override
get the size of the source states
Definition: multiplicativeAlignment.cpp:66
void * get() const
Definition: sampleDevice.h:264
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: flattenConcat.cpp:210
int embSkipLayerNorm2(cudaStream_t stream, int ld, int B, int S, const int *inputIds, const int *tokenIds, const int *cuSeqlens, const float *beta, const float *gamma, const T *wordEmb, const T *posEmb, const T *tokEmb, T *output)
Conditional type.
Definition: ieee_half.h:296
std::vector< float > mBatch
Data for the batch.
Definition: BatchStream.h:379
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: sampleUffPluginV2Ext.cpp:658
void safeFree(T *ptr)
Definition: regionPlugin.cpp:31
Definition: sampleDevice.h:289
size_t mPosVocabSize
Definition: embLayerNormVarSeqlenPlugin.h:95
def main()
Definition: test_infer.py:110
def test_extend_1_args_0_rv(self)
Definition: test_func.py:69
static void help(std::ostream &out)
Definition: sampleOptions.cpp:1278
void operator()(void *ptr) const
Definition: buffers.h:191
std::string name
Definition: sampleReporting.h:160
_learn_amax
Definition: tensor_quant.py:88
static const std::vector< std::string > CLASS_NAMES
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:86
@ kFLOAT32
FP32 field type.
~SplitPluginCreator()
Definition: split.h:140
nvinfer1::IErrorRecorder * getErrorRecorder() const override
get the ErrorRecorder assigned to this interface.
Definition: caffeParser.h:60
nvinfer1::ILayer * parseConcat(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &, BlobNameToTensor &tensors)
Definition: parseConcat.cpp:23
long lrint(half arg)
Nearest integer using half's internal rounding mode.
Definition: ieee_half.h:2600
nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: qkvToContextInt8InterleavedPlugin.cpp:159
int batchSize
Definition: sampleMovieLens.cpp:525
std::shared_ptr< CudaBind< float > > mAnchorBoxesDevice
Definition: proposalLayerPlugin.h:106
def test_input_fake_quant(self)
Definition: quant_pooling_test.py:297
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: batchedNMSPlugin.cpp:414
CUresult(* _cuGetErrorName)(CUresult, const char **)
Definition: cudaDriverWrapper.h:70
void clear()
Definition: fused_multihead_attention_v2.h:73
Definition: backend/base/loader.py:1
std::string mNameSpace
Definition: resizeNearestPlugin.h:93
nvinfer1::DimsHW sortClassLabelDims
Definition: maskRCNNKernels.h:168
Definition: NvInferRuntime.h:375
~CaffeParser() override
Definition: caffeParser.cpp:32
bool infer()
Runs inference using TensorRT on a random image.
Definition: sampleDynamicReshape.cpp:321
int x
Definition: maskRCNNKernels.h:258
virtual void addToModel(nvinfer1::INetworkDefinition *network, nvinfer1::ITensor *attentionKeys, nvinfer1::ITensor *queryStates, nvinfer1::ITensor **alignmentScores)=0
add the alignment scores calculation to the network
def str_from_config(config)
Definition: trt/util.py:266
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: proposalLayerPlugin.cpp:153
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: embLayerNormVarSeqlenPlugin.cpp:357
TestAtom(TestAtom &&)=default
@ kMAX
Maximum of the two elements.
int initialize() override
Initialize the layer for execution.
Definition: pyramidROIAlignPlugin.cpp:93
~InstanceNormalizationPlugin() override
Definition: instanceNormalizationPlugin.cpp:137
def load_checkpoint(model, optimizer, epoch, config, amp_run, filepath, local_rank)
Definition: train.py:229
The SampleMLP class implements the MNIST API sample.
Definition: sampleMLP.cpp:57
nvinfer1::Dims toDims(const std::vector< int > &vec)
Definition: sampleUtils.h:151
std::shared_ptr< CudaBind< float > > mRegWeightDevice
Definition: generateDetectionPlugin.h:105
bool verifyOutput(const samplesCommon::BufferManager &buffers, const std::string &outputTensorName, int groundTruthDigit) const
Verifies that the output is correct and prints it.
Definition: sampleUffMNIST.cpp:225
bool build()
Builds the network engine.
Definition: sampleCharRNN.cpp:284
const char * PROPOSALLAYER_PLUGIN_NAME
Definition: proposalLayerPlugin.cpp:33
bool mStarted
Definition: pluginLogging.h:302
def is_valid_shape_override(new_shape, original_shape)
Definition: Polygraphy/polygraphy/util/misc.py:147
Compression and Serialization.
Definition: Polygraphy/polygraphy/util/misc.py:175
virtual Weights getWeights() const =0
Get the weights for the layer.
max_wav_value
Definition: waveglow/data_function.py:31
The NMSParameters are used by the BatchedNMSPlugin for performing the non_max_suppression operation o...
Definition: NvInferPluginUtils.h:247
TextWriter(std::shared_ptr< std::ostream > textOnput, Vocabulary::ptr vocabulary)
Definition: textWriter.cpp:24
std::vector< size_t > mCopySize
Definition: flattenConcat.h:99
unsigned int fused_multihead_attention_fp16_384_64_kernel_sm80_cu_o_len
Definition: fused_multihead_attention_fp16_384_64_kernel.sm80.cpp:1664
void setFailurePercentage(float f)
Definition: sampleConfig.h:291
nvinfer1::Dims mInputDims
The dimensions of the input to the network.
Definition: sampleINT8API.cpp:110
Definition: samples/common/common.h:652
const char * MULTILEVELPROPOSEROI_PLUGIN_VERSION
Definition: multilevelProposeROIPlugin.cpp:34
bool mEnableVariableLen
Definition: bert_infer.h:343
float mRpnStdScaling
Definition: proposalPlugin.h:113
key
Definition: Tacotron2/inference.py:167
host_output_buffers
Definition: backend/trt/runner.py:100
virtual ~ISliceLayer()
Definition: NvInfer.h:4029
half(detail::binary_t, detail::uint16 bits)
Constructor.
Definition: samples/common/half.h:1559
def test_save_tensorboard(self)
Definition: tf/test_loader.py:68
thrust::device_vector< int > _d_segment_offsets
Definition: split.h:44
Definition: sampleUffMaskRCNN.cpp:310
void parse(Arguments &arguments) override
Definition: sampleOptions.cpp:406
static std::vector< PluginField > mPluginAttributes
Definition: priorBoxPlugin.h:117
def init_quantizer(self, quant_desc_input, quant_desc_weight, num_layers=None)
Definition: _utils.py:56
Definition: license_test.py:26
nvinfer1::Dims getDims() const override
Definition: BatchStream.h:279
void write(char *&buffer, const T &val) const
Definition: sampleUffPluginV2Ext.cpp:580
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: regionPlugin.cpp:414
def try_call(func, *args, **kwargs)
Definition: Polygraphy/polygraphy/util/misc.py:267
text_cleaners
Definition: tacotron2/data_function.py:34
void accumulateBLEU(const std::vector< Segment_t > &referenceSamples, const std::vector< Segment_t > &outputSamples, int maxOrder, size_t &referenceLength, size_t &translationLength, std::vector< size_t > &matchesByOrder, std::vector< size_t > &possibleMatchesByOrder)
Definition: bleuScoreWriter.cpp:119
Definition: tf/test_loader.py:34
static const char * EMB_LAYER_NORM_VAR_SEQLEN_VERSION
Definition: embLayerNormVarSeqlenPlugin.cpp:47
mode
Definition: modules/quant_rnn.py:45
bool spin
Definition: sampleOptions.h:156
int initialize() override
Initialize the layer for execution.
Definition: batchedNMSPlugin.cpp:90
std::string mNamespace
Definition: proposalPlugin.h:108
const char * getPluginVersion() const override
Return the plugin version.
Definition: embLayerNormVarSeqlenPlugin.cpp:430
def test_import_node(self)
Definition: test_importers.py:63
virtual void destroy()=0
Destroy this INetworkDefinition object.
def get_output_metadata(network)
Definition: trt/util.py:88
def get_last_checkpoint_filename(output_dir, model_name)
Definition: train.py:219
def parse_args(parser)
Definition: inference_trt.py:36
int y
Definition: maskRCNNKernels.h:257
void initialize(int sampleCount, int *maxOutputSequenceLengths)
Definition: beamSearchPolicy.cpp:37
T stringToValue(const std::string &option)
Definition: sampleOptions.cpp:55
virtual const char * getModelFileName() const
Get the Model FileName.
Definition: parserOnnxConfig.h:87
def infer_impl(self, feed_dict)
Definition: backend/onnx/runner.py:50
std::string mPluginNamespace
Definition: flattenConcat.h:104
@ kCOUNT
Tensor is scalar of type kINT32 that contains the trip count.
def nested_dup_names()
Definition: onnx_models.py:185
def activate_impl(self)
Definition: backend/trt/runner.py:59
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: gridAnchorPlugin.cpp:198
int initialize() override
Initialize the layer for execution.
Definition: flattenConcat.cpp:121
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: nvFasterRCNNPlugin.cpp:349
def forward(self, mel_outputs)
Definition: export_tacotron2_onnx.py:81
float mIOUThreshold
Definition: multilevelProposeROIPlugin.h:99
virtual ITensor * getOutput(int32_t index) const =0
Get the output tensor specified by the given index.
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: pyramidROIAlignPlugin.cpp:112
def test_against_unquantized(self)
Definition: quant_conv_transposed_test.py:307
static const float DETECTION_REG_WEIGHTS[]
Definition: tlt_mrcnn_config.h:37
def get_output_metadata(graph, layerwise=False)
Definition: tf/util.py:94
def load_vocab(vocab_file)
Definition: tokenization.py:120
def test_fake_quant_per_channel(self)
Definition: quant_instancenorm_test.py:79
def findAndRemove(profile, name)
Definition: profiler.py:70
max_decoder_steps
Definition: demo/Tacotron2/tacotron2/model.py:248
const char * getModelFileName() const
Get the Model FileName.
Definition: sampleConfig.h:104
def test_layer_with_arrays(self)
Definition: test_ir.py:532
early_stopping
Definition: demo/Tacotron2/tacotron2/model.py:252
virtual void addToModel(nvinfer1::INetworkDefinition *network, int maxInputSequenceLength, nvinfer1::ITensor *inputEmbeddedData, nvinfer1::ITensor *actualInputSequenceLengths, nvinfer1::ITensor **inputStates, nvinfer1::ITensor **memoryStates, nvinfer1::ITensor **lastTimestepStates)=0
add the memory and last timestep states to the network lastTimestepHiddenStates is the pointer to the...
static const std::vector< std::string > CLASS_NAMES
Definition: tlt_mrcnn_config.h:86
alignment scores from Luong attention mechanism
Definition: multiplicativeAlignment.h:31
Definition: tools/Polygraphy/polygraphy/backend/cntk/__init__.py:1
def whitespace_tokenize(text)
Definition: tokenization.py:151
void * qkv_ptr
Definition: fused_multihead_attention_v2.h:28
bool readBinaryProto(trtcaffe::NetParameter *net, const char *file, size_t bufSize)
Definition: readProto.h:37
def test_simple_build(self)
Definition: integration_test.py:46
def test_multithreaded_runners_from_engine(self)
Definition: trt/test_runner.py:66
void record(const TrtCudaStream &stream)
Definition: sampleDevice.h:134
def test_infer_raises_if_runner_inactive()
Definition: base/test_runner.py:8
@ kINVALID_STATE
TensorRT was put into a bad state by incorrect sequence of function calls.
def decompress(compressed)
Definition: Polygraphy/polygraphy/util/misc.py:192
virtual size_t getSerializationSize() override
Find the size of the serialization buffer required.
Definition: samples/opensource/samplePlugin/fcPlugin.h:204
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: multilevelProposeROIPlugin.cpp:436
nvcaffeparser1::IPluginFactory * mPluginFactory
Definition: caffeParser.h:87
size_t type2size(nvinfer1::DataType type)
Definition: samples/opensource/samplePlugin/fcPlugin.h:224
virtual void setDLACore(int32_t dlaCore) noexcept=0
Set the DLA core that the deserialized engine must execute on.
virtual ~IPoolingLayer()
Definition: NvInfer.h:1862
detail::enable< half &, T >::type operator+=(T rhs)
Arithmetic assignment.
Definition: samples/common/half.h:1441
std::unordered_map< std::string, AlgorithmCacheItem > choiceMap
Definition: sampleAlgorithmSelector.cpp:243
def compare_accuracy(run_results, fail_fast=False, comparisons=None, compare_func=None)
Definition: comparator/comparator.py:214
def forward(self, attention_hidden_state, memory, processed_memory, attention_weights_cat, mask)
Definition: demo/Tacotron2/tacotron2/model.py:85
T type
Definition: ieee_half.h:385
def load_commands_from_readme(readme)
Definition: onnx-graphsurgeon/tests/test_examples.py:52
std::vector< float > mResizedKernelWeights
Definition: slpEmbedder.h:55
int mNumHeads
Definition: qkvToContextInt8InterleavedPlugin.h:86
quant_linear
Definition: print_test.py:50
Severity mReportableSeverity
Definition: samples/common/logging.h:441
def free(self)
Definition: cuda.py:95
def _quant(self, input)
Definition: quant_conv.py:73
static void reportTestEnd(const TestAtom &testAtom, TestResult result)
Report that a test has ended.
Definition: samples/common/logging.h:340
std::string fileName
Definition: samples/common/common.h:654
bool int8
Definition: sampleOptions.h:115
expr log10(half arg)
Common logorithm.
Definition: ieee_half.h:2354
def test_segfault_does_not_hang(self)
Definition: test_comparator.py:110
const char * getPluginVersion() const override
Return the plugin version.
Definition: regionPlugin.cpp:388
static expr exp(float arg)
Exponential implementation.
Definition: samples/common/half.h:1767
nvinfer1::DimsHW concatedScoreDims
Definition: maskRCNNKernels.h:195
virtual void configurePlugin(const PluginTensorDesc *in, int32_t nbInput, const PluginTensorDesc *out, int32_t nbOutput)=0
Configure the layer.
LikelihoodCombinationOperator::ptr getLikelihoodCombinationOperator() const override
Definition: softmaxLikelihood.cpp:98
constexpr int32_t EnumMax< SliceMode >()
Maximum number of elements in SliceMode enum.
Definition: NvInfer.h:3889
virtual void setErrorRecorder(IErrorRecorder *recorder) noexcept=0
Set the ErrorRecorder for this interface.
EmbLayerNormVarSeqlenPlugin()=delete
std::string loadEngine
Definition: argsParser.h:88
virtual ~IBuilderConfig()
Definition: NvInfer.h:6736
bool mError
Definition: blobNameToTensor.h:68
virtual ~ILRNLayer()
Definition: NvInfer.h:2060
Int8EntropyCalibrator2(BatchStream &stream, int firstBatch, const char *networkName, const char *inputBlobName, bool readCache=true)
Definition: frcnnUtils.h:510
static std::list< DumpTensorPlugin::ptr > mPlugins
Definition: debugUtil.h:80
virtual ~ILogger()
Definition: NvInferRuntimeCommon.h:1101
int h
Definition: fused_multihead_attention.h:79
def __repr__(self)
Definition: tools/pytorch-quantization/pytorch_quantization/calib/calibrator.py:56
virtual void * data() const noexcept=0
A pointer to the raw data that is owned by the library.
virtual const IBlobNameToTensor * parse(const char *deploy, const char *model, nvinfer1::INetworkDefinition &network, nvinfer1::DataType weightType)=0
Parse a prototxt file and a binaryproto Caffe model to extract network definition and weights associa...
std::vector< std::string > inputImages
Definition: frcnnUtils.h:63
int scoresSize
Definition: batchedNMSPlugin.h:69
float getTotalTime() const
Definition: sampleReporting.h:182
bool fp16
Definition: sampleOptions.h:114
LikelihoodCombinationOperator()=default
virtual bool dynamicRangeIsSet() const =0
Query whether dynamic range is set.
ICaffeParser * createCaffeParser()
Creates a ICaffeParser object.
Definition: NvCaffeParser.cpp:32
const char * getPluginNamespace() const override
Return the namespace of the plugin creator object.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:344
int avgs
Definition: sampleOptions.h:172
void check_valid_inputs(const nvinfer1::Dims *inputs, int nbInputDims)
Definition: specialSlicePlugin.cpp:137
_calib_amax
Definition: max.py:42
max
Definition: backend/trt/loader.py:247
onnx.ValueInfoProto export_value_info_proto(Tensor tensor, bool do_type_check)
Definition: onnx_exporter.py:39
Application-implemented class for controlling allocation on the GPU.
Definition: NvInferRuntimeCommon.h:1033
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: nmsPlugin.cpp:202
def __init__(self, graph, outputs=None)
Definition: backend/tf/loader.py:271
int mFileCount
Definition: BatchStream.h:376
virtual DataType getType() const =0
Get the data type of a tensor.
int initialize() override
Initialize the layer for execution.
Definition: batchedNMSPlugin.cpp:85
def __init__(self, host_mem, device_mem)
Definition: backend/trt_legacy.py:180
opset
Definition: backend/onnx/loader.py:68
static nvinfer1::PluginFieldCollection mFC
Definition: skipLayerNormInt8InterleavedPlugin.h:114
std::string outputRegName
Definition: sampleUffFasterRCNN.cpp:64
std::map< std::string, std::pair< nvinfer1::Dims, nvinfer1::Weights > > mWeightMap
The weight name to weight value map.
Definition: sampleMLP.cpp:90
constexpr int32_t EnumMax< PluginType >()
Maximum number of elements in PluginType enum.
Definition: NvInferPluginUtils.h:53
nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
std::string toLower(const std::string &inp)
Definition: samples/common/common.h:494
Definition: sampleConfig.h:26
bool isSupported(DataType dataType)
Checks if the platform supports the data type.
Definition: sampleINT8.cpp:178
Definition: tacotron2/loss_function.py:20
Definition: demo/Tacotron2/tacotron2/model.py:29
std::unordered_map< const unsigned char *, CUmodule > mModules
Definition: fused_multihead_attention.h:281
void resize(const nvinfer1::Dims &dims)
Overload of resize that accepts Dims.
Definition: buffers.h:161
tensors
Definition: test_ir.py:223
int num
Definition: regionPlugin.h:93
std::vector< std::string > inputImages
Definition: sampleUffFasterRCNN.cpp:67
virtual void setPrintLayerInfo(bool)=0
Set whether the layer information will be printed.
virtual ~IConvolutionLayer()
Definition: NvInfer.h:1368
unsigned char fused_multihead_attention_v2_int8_128_64_kernel_sm80_cubin[]
Definition: fused_multihead_attention_v2_int8_128_64_kernel.sm80.cpp:19
Definition: backend/trt/loader.py:60
def __init__(self, path, shapes, outputs)
Definition: backend/trt_legacy.py:36
TestResult
Represents the state of a given test.
Definition: samples/common/logging.h:218
static const int mFeatureMapCount
Definition: pyramidROIAlignPlugin.h:95
int getId(const std::string &token) const
get the ID of the token
Definition: vocabulary.cpp:42
__class__
Definition: tensor.py:69
int mMaxBatches
Definition: BatchStream.h:148
projects 1-hot vectors (represented as a vector with indices) into dense embedding space
Definition: embedder.h:32
virtual bool isPluginExt(const char *layerName)=0
A user implemented function that determines if a layer configuration is provided by an IPluginExt.
samplesCommon::ManagedBuffer mOutput
Host buffer for the ouptut.
Definition: sampleDynamicReshape.cpp:91
~TrtCudaStream()
Definition: sampleDevice.h:73
Definition: backend/trt/loader.py:111
def create_stream(self)
Definition: cuda.py:45
static bool signbit(half arg)
Sign bit implementation.
Definition: samples/common/half.h:2378
int max
Definition: sampleUffMaskRCNN.cpp:75
Definition: test_importers.py:32
int mBatchDim
Definition: lReluPlugin.h:73
const char * DETECTIONLAYER_PLUGIN_NAME
Definition: detectionLayerPlugin.cpp:29
expr cos(half arg)
Cosine function.
Definition: ieee_half.h:2424
def forward(self, x, input_lengths)
Definition: demo/Tacotron2/tacotron2/model.py:200
def griffin_lim(magnitudes, stft_fn, n_iters=30)
Definition: audio_processing.py:75
expr acosh(half arg)
Hyperbolic area cosine.
Definition: ieee_half.h:2501
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: flattenConcat.cpp:133
int val
Definition: getopt.h:79
def set_output_name(layer, prefix, name, out_idx=0)
Definition: builder_varseqlen.py:114
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: flattenConcat.cpp:311
def _compute_amax_entropy(self, stride, start_bin)
Definition: histogram.py:92
size_t getWorkspaceSize(const nvinfer1::PluginTensorDesc *inputs, int nbInputs, const nvinfer1::PluginTensorDesc *outputs, int nbOutputs) const override
Definition: instanceNormalizationPlugin.cpp:166
def __call__(self, batch)
Definition: tacotron2/data_function.py:89
~Alignment() override=default
int32_t RefCount
A typedef of a 32bit integer for reference counting.
Definition: NvInferRuntimeCommon.h:1244
virtual ~EntropyCalibratorImpl()
Definition: EntropyCalibrator.h:44
def identity_builder_network()
Definition: trt/test_loader.py:41
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: cropAndResizePlugin.cpp:131
static const int mFeatureMapCount
Definition: multilevelCropAndResizePlugin.h:95
letter
Definition: Polygraphy/polygraphy/logger/logger.py:131
def get_output_metadata(graph)
Definition: onnx/util.py:164
expr log2(half arg)
Binary logorithm.
Definition: ieee_half.h:2368
def _arpabet_to_sequence(text)
Definition: demo/Tacotron2/tacotron2/text/__init__.py:69
float mIOUThreshold
Definition: multilevelProposeROIPlugin.h:142
int32_t nbFields
Definition: NvUffParser.h:89
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: nmsPlugin.cpp:67
attention_layer
Definition: demo/Tacotron2/tacotron2/model.py:262
_graph
Definition: backend/tf/loader.py:286
std::string mReferenceFilename
Definition: sampleConfig.h:41
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleDynamicReshape.cpp:48
int device
Definition: sampleOptions.h:136
segment_length
Definition: waveglow/data_function.py:37
virtual IFullyConnectedLayer * addFullyConnected(ITensor &input, int32_t nbOutputs, Weights kernelWeights, Weights biasWeights)=0
Add a fully connected layer to the network.
@ kSAFE_DLA
Safety restricted capability, TensorRT flow that can only run on DLA devices via NvMediaDLA APIs.
Definition: quant_pooling_test.py:149
static const std::string MODEL_NAME
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:170
DataType mPrecision
Definition: batchedNMSPlugin.h:73
int max
Definition: samples/common/common.h:655
unsigned int fused_multihead_attention_fp16_128_64_kernel_sm75_cu_o_len
Definition: fused_multihead_attention_fp16_128_64_kernel.sm75.cpp:2702
SampleINT8API(const SampleINT8APIParams ¶ms)
Definition: sampleINT8API.cpp:83
def main(cmdline_args)
Definition: classification_flow.py:182
int stride
Definition: reorgPlugin.h:88
void dumpBindingDimensions(int binding, const nvinfer1::IExecutionContext &context, std::ostream &os) const
Definition: sampleUtils.h:371
def test_topologically_sort(self, toposort_test_case)
Definition: test_ir.py:618
void transposeWeights(nvinfer1::Weights &wts, int hiddenSize)
Transpose weights.
Definition: sampleMLP.cpp:461
virtual int32_t getNbDLACores() const noexcept=0
Returns number of DLA hardware cores accessible.
Convenience object used to facilitate use of C++ stream syntax when logging messages....
Definition: samples/common/logging.h:132
nvinfer1::DimsCHW getCHW(const nvinfer1::Dims &d)
Definition: parserUtils.h:108
float mThresh
Definition: pyramidROIAlignPlugin.h:99
int32_t nmsMaxOut
Definition: NvInferPluginUtils.h:122
Object used to store and query data extracted from a binaryproto file using the ICaffeParser.
Definition: NvCaffeParser.h:71
def onnx_to_trt_name(onnx_name)
Definition: builder.py:440
const char * getPluginNamespace() const override
Return the namespace of the plugin creator object.
Definition: plugin.h:68
Logger gLogger
Definition: logger.cpp:22
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: priorBoxPlugin.cpp:209
size_t detectionInferenceWorkspaceSize(bool shareLocation, int N, int C1, int C2, int numClasses, int numPredsPerClass, int topK, DataType DT_BBOX, DataType DT_SCORE)
Definition: kernel.cpp:39
def test_export(self)
Definition: test_api.py:35
stft
Definition: denoiser.py:30
std::map< std::string, Record > mProfile
Definition: samples/common/common.h:222
def infer(self, feed_dict)
Definition: backend/base/runner.py:104
SampleCharRNNParams mParams
Definition: sampleCharRNN.cpp:187
static PluginFieldCollection mFC
Definition: multilevelCropAndResizePlugin.h:123
Definition: backend/trt/loader.py:299
~LogStreamConsumerBuffer()
Definition: samples/common/logging.h:51
Definition: tools/pytorch-quantization/pytorch_quantization/optim/__init__.py:1
virtual DimsExprs getOutputDimensions(int32_t outputIndex, const DimsExprs *inputs, int32_t nbInputs, IExprBuilder &exprBuilder)=0
Get expressions for computing dimensions of an output tensor from dimensions of the input tensors.
void operator()(softmaxTree *smTree) const
Definition: regionPlugin.cpp:48
audiopaths_and_text
Definition: waveglow/data_function.py:30
constexpr const char * SPLIT_PLUGIN_VERSION
Definition: split.h:30
std::vector< GridAnchorParameters > mParam
Definition: gridAnchorPlugin.h:96
void serialize(void *buffer) const override
Serialize the layer.
Definition: coordConvACPlugin.cpp:94
def clear(self)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:123
Definition: tensor_quantizer_test.py:36
float * getLabels() override
Definition: BatchStream.h:264
std::ostream & printPrecision(std::ostream &os, const BuildOptions &options)
Definition: sampleOptions.cpp:372
void * getDeviceBuffer() const
Definition: sampleDevice.h:323
outputs
Definition: onnx_models.py:38
Plugin class for user-implemented layers.
Definition: NvInferRuntimeCommon.h:716
void setClipParam(bool clip)
Definition: batchedNMSPlugin.cpp:454
def test_mark_layerwise(self, identity_identity_network)
Definition: trt/test_loader.py:121
int maxBatch
Definition: sampleOptions.h:108
static T cast(expr arg)
Definition: samples/common/half.h:2666
def forward(self, forward_input)
Definition: demo/Tacotron2/waveglow/model.py:186
float iouThreshold
Definition: NvInferPluginUtils.h:125
void constructNetwork(SampleUniquePtr< nvcaffeparser1::ICaffeParser > &parser, SampleUniquePtr< nvinfer1::IBuilder > &builder, SampleUniquePtr< nvinfer1::INetworkDefinition > &network, SampleUniquePtr< nvinfer1::IBuilderConfig > &config)
Parses a Caffe model for FasterRCNN and creates a TensorRT network.
Definition: sampleFasterRCNN.cpp:181
@ kANCHORGENERATOR
SSD Grid Anchor Generator.
void copyOutputToHost()
Copy the contents of output device buffers to output host buffers synchronously.
Definition: buffers.h:403
virtual IActivationLayer * addActivation(ITensor &input, ActivationType type)=0
Add an activation layer to the network.
int mNbGroups
Definition: groupNormalizationPlugin.h:99
bool build()
Function builds the network engine.
Definition: sampleMLP.cpp:142
int getMemoryStatesSize() override
get the size of the memory state vector
Definition: lstmEncoder.cpp:100
def test_register(self)
Definition: test_ir.py:461
const char * getPluginVersion() const override
Return the plugin version.
Definition: proposalPlugin.cpp:155
const char * GRID_ANCHOR_PLUGIN_VERSION
Definition: gridAnchorPlugin.cpp:30
Definition: comparator/data_loader.py:198
int mKeepTopK
Definition: detectionLayerPlugin.h:95
constexpr int32_t EnumMax()
Forward declare IGpuAllocator for use in other interfaces.
def unsigned(self)
Definition: tensor_quantizer.py:114
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: proposalPlugin.cpp:294
void * getHostBuffer(const std::string &tensorName) const
Returns the host buffer corresponding to tensorName.
Definition: buffers.h:304
nvinfer1::DimsHW sortClassLabelDims
Definition: maskRCNNKernels.h:133
def build_engine(model_file, shapes, max_ws=512 *1024 *1024, fp16=False)
Definition: trt_utils.py:92
@ kTANH
Single gate RNN w/ TANH activation function.
def evaluating(model)
Definition: train.py:249
int copyFromHost(char *dstHostBuffer, const void *source, int count) const
Definition: nvFasterRCNNPlugin.cpp:208
void configurePlugin(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, const DataType *inputTypes, const DataType *outputTypes, const bool *inputIsBroadcast, const bool *outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override
Definition: flattenConcat.cpp:227
def get_alignment_energies(self, query, processed_memory, attention_weights_cat)
Definition: demo/Tacotron2/tacotron2/model.py:63
samplesCommon::OnnxSampleParams mParams
The parameters for the sample.
Definition: sampleOnnxMNIST.cpp:69
def __call__(self)
Definition: backend/trt/loader.py:123
IPluginV2 * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: lReluPlugin.cpp:155
def backward(ctx, grad_output)
Definition: functional.py:43
PluginField(const char *name_=nullptr, const void *data_=nullptr, const PluginFieldType type_=PluginFieldType::kUNKNOWN, int32_t length_=0)
Definition: NvInferRuntimeCommon.h:867
def __call__(self)
Definition: backend/tf/loader.py:374
def build_engine(batch_size, workspace_size, sequence_length, config, weights_dict, squad_json, vocab_file, calibrationCacheFile, calib_num)
Definition: builder_varseqlen.py:546
Definition: tools/pytorch-quantization/pytorch_quantization/utils/__init__.py:1
int enqueue(const PluginTensorDesc *inputDesc, const PluginTensorDesc *outputDesc, const void *const *inputs, void *const *outputs, void *workspace, cudaStream_t stream) override
Execute the layer.
Definition: batchedNMSPlugin.cpp:229
size_t getWorkspaceSize(const nvinfer1::PluginTensorDesc *inputs, int nbInputs, const nvinfer1::PluginTensorDesc *outputs, int nbOutputs) const override
Definition: qkvToContextInt8InterleavedPlugin.cpp:152
int32_t userId
Definition: sampleMovieLens.cpp:51
bool mClipBoxes
Definition: batchedNMSPlugin.h:117
virtual IElementWiseLayer * addElementWise(ITensor &input1, ITensor &input2, ElementWiseOperation op)=0
Add an elementwise layer to the network.
def copy(self)
Definition: tensor.py:210
virtual void setFullTextFileName(const char *fullTextFileName)=0
Set the File Name of the Network Description as a Text File, including the weights.
num_inputs
Definition: demo/BERT/helpers/calibrator.py:40
virtual int32_t getNbLayers() const =0
Get the number of layers in the network.
~MultilevelCropAndResizePluginCreator()
Definition: multilevelCropAndResizePlugin.h:110
bool teardown()
Used to clean up any state created in the sample class.
Definition: sampleMNIST.cpp:345
cudaEvent_t get() const
Definition: sampleDevice.h:129
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: sampleUffPluginV2Ext.cpp:391
mask_padding
Definition: demo/Tacotron2/tacotron2/model.py:590
const char * getPluginName() const override
Return the plugin name.
Definition: instanceNormalizationPlugin.cpp:335
void operator()(T *t)
Definition: demo/BERT/infer_c/common.h:169
Vocabulary::ptr mVocabulary
Definition: bleuScoreWriter.h:53
std::vector< std::string > mIdToToken
Definition: vocabulary.h:73
PriorBox(PriorBoxParameters param, int H=0, int W=0)
Definition: priorBoxPlugin.cpp:40
line_info
Definition: Polygraphy/polygraphy/logger/logger.py:133
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: specialSlicePlugin.cpp:212
float operator-(const TrtCudaEvent &e) const
Definition: sampleDevice.h:145
bool getTF32() const
Definition: sampleConfig.h:94
def mel_spectrogram(self, y)
Definition: layers.py:78
Definition: gridAnchorPlugin.h:30
nvinfer1::Dims mDims
Definition: debugUtil.h:69
_input_quantizers
Definition: _utils.py:83
def __call__(self)
Definition: backend/onnx/loader.py:129
TRTOption defines a command line option.
Definition: getOptions.h:34
bool processInput(const samplesCommon::BufferManager &buffers)
Reads the input and stores the result in a managed buffer.
Definition: sampleOnnxMNIST.cpp:238
Definition: test_ir.py:30
void * bnScale
Definition: groupNormalizationPlugin.h:105
void * mDeviceBias
Definition: samples/opensource/samplePlugin/fcPlugin.h:313
@ kELEMENTWISE
Elementwise layer.
def __getitem__(self, index)
Definition: waveglow/data_function.py:65
LogStreamConsumer gLogVerbose
Definition: logger.cpp:23
half floor(half arg)
Nearest integer not greater than half value.
Definition: ieee_half.h:2558
std::string getToken(int id) const
get token by ID
Definition: vocabulary.cpp:50
Plugin creator class for user implemented layers.
Definition: NvInferRuntimeCommon.h:890
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: proposalLayerPlugin.cpp:368
std::string calibration
Definition: sampleOptions.h:122
CUresult cuGetErrorName(CUresult error, const char **pStr) const
Definition: cudaDriverWrapper.cpp:71
~Projection() override=default
def propagate_dtype(outputs, dtype)
Definition: 07_creating_a_model_with_the_layer_api/generate.py:25
int mNbClasses
Definition: detectionLayerPlugin.h:127
bool rerun
Definition: sampleOptions.h:160
void allocateBindings(const int maxBatchSize)
Definition: bert_infer.h:108
def test_import_constant_tensor(self)
Definition: test_importers.py:44
sigma
Definition: waveglow/loss_function.py:22
bool constructNetwork(SampleUniquePtr< nvinfer1::IBuilder > &builder, SampleUniquePtr< nvinfer1::INetworkDefinition > &network, SampleUniquePtr< nvuffparser::IUffParser > &parser)
Definition: sampleUffMaskRCNN.cpp:388
use_qat
Definition: builder_varseqlen.py:100
static std::string severityPrefix(Severity severity)
Definition: pluginLogging.h:167
def match_parameters(model, patterns)
Definition: helper.py:24
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: resizeNearestPlugin.cpp:93
int mElemsPerRow
Definition: debugUtil.h:71
Definition: quant_pooling.py:103
int enqueue(int batch_size, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: specialSlicePlugin.cpp:162
const char * getPluginType() const override
Return the plugin type.
Definition: multilevelProposeROIPlugin.cpp:211
is_calib_mode
Definition: builder.py:103
def __init__(self, min, opt, max)
Definition: backend/trt/loader.py:236
shapes
Definition: backend/trt_legacy.py:38
def test_import_graph_with_dim_param(self)
Definition: test_importers.py:119
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleOnnxMnistCoordConvAC.cpp:81
x0
Definition: 03_isolating_a_subgraph/generate.py:26
int32_t nbDims
The number of dimensions.
Definition: NvInferRuntimeCommon.h:198
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: nvFasterRCNNPlugin.cpp:173
nvinfer1::ILayer * parseCrop(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &, BlobNameToTensor &tensors)
Definition: parseCrop.cpp:23
const char * GENERATEDETECTION_PLUGIN_VERSION
Definition: generateDetectionPlugin.cpp:31
nvinfer1::ITensor *& operator[](const std::string &name)
Definition: blobNameToTensor.h:46
def _generate_name(self, prefix)
Definition: graph.py:391
@ kDEBUG
Enable debugging of layers via synchronizing after every layer.
static const float RPN_STD_SCALING
Definition: proposalPlugin.cpp:34
nvinfer1::DimsHW sortedScoreDims
Definition: maskRCNNKernels.h:197
def StackedRNN(inners, num_layers, lstm=False, dropout=0, train=True)
Definition: _functions/quant_rnn.py:88
void * getBuffer(const bool isHost, const std::string &tensorName) const
Definition: buffers.h:427
Definition: backend/trt_legacy.py:48
int open(int flag, mode_t mode)
Definition: sampleMovieLensMPS.cpp:142
int getSize() const
get the number of elements in the vocabulary
Definition: vocabulary.cpp:56
Definition: quant_conv_transposed_test.py:334
uint64_t hashID(unsigned int s, unsigned int d) const
Definition: fused_multihead_attention.h:194
cudaStream_t mStream
Definition: samples/common/common.h:836
A fully connected layer in a network definition. This layer expects an input tensor of three or more ...
Definition: NvInfer.h:1560
static const std::string MODEL_INPUT
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:171
@ kPASSED
The test passed.
name
Definition: graph.py:118
_input_quantizer
Definition: _utils.py:80
virtual nvinfer1::DimsNCHW getDimensions()=0
Definition: checkMacrosPlugin.h:47
const char * getReferenceFileName() const
Definition: sampleConfig.h:241
SampleBuffer(nvinfer1::Dims dims, int dataWidth, TensorFormat format)
Definition: sampleReformatFreeIO.cpp:143
static std::string genCmdlineString(int argc, char const *const *argv)
generate a command line string from the given (argc, argv) values
Definition: samples/common/logging.h:427
def test_unpack_args(case)
Definition: test_misc.py:86
constexpr int defaultStreams
Definition: sampleOptions.h:45
Weights deserializeToDevice(const char *&hostBuffer, size_t count)
Definition: priorBoxPlugin.cpp:228
int mPreNMSTopK
Definition: multilevelProposeROIPlugin.h:139
Definition: backend/tf/loader.py:161
num_attention_heads
Definition: builder_varseqlen.py:93
BeamSearchPolicy(int endSequenceId, LikelihoodCombinationOperator::ptr likelihoodCombinationOperator, int beamWidth)
Definition: beamSearchPolicy.cpp:29
xy_t(int y_, int x_)
Definition: maskRCNNKernels.h:265
def get_active_profile_bindings(context)
Definition: trt/util.py:432
int avgTiming
Definition: sampleOptions.h:111
cudaStream_t get() const
Definition: sampleDevice.h:78
detail::enable< half &, T >::type operator*=(T rhs)
Arithmetic assignment.
Definition: samples/common/half.h:1461
Definition: sampleUffMaskRCNN.cpp:304
def test_cleanup_multi_tier(self)
Definition: test_ir.py:625
bool verifyOutput(const samplesCommon::BufferManager &buffers, const std::string &outputTensorName, int groundTruthDigit) const
Verifies that the output is correct and prints it.
Definition: sampleAlgorithmSelector.cpp:476
def test_fake_quant_per_channel_bias(self)
Definition: quant_conv_test.py:410
T half2int(uint16 value)
Convert half-precision floating point to integer.
Definition: ieee_half.h:995
def __init__(self, path_components, artifact_names=[])
Definition: Polygraphy/tests/test_examples.py:55
SampleUffMNIST(const samplesCommon::UffSampleParams ¶ms)
Definition: sampleUffMNIST.cpp:59
static const char * QKV_TO_CONTEXT_INTERLEAVED_PLUGIN_NAME
Definition: qkvToContextInt8InterleavedPlugin.cpp:40
static expr minus(float x, float y)
Subtraction implementation.
Definition: samples/common/half.h:1598
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: regionPlugin.cpp:425
size_t getWorkspaceSize(const nvinfer1::PluginTensorDesc *inputs, int nbInputs, const nvinfer1::PluginTensorDesc *outputs, int nbOutputs) const override
Definition: skipLayerNormInt8InterleavedPlugin.cpp:155
friend std::istream & operator>>(std::istream &input, ComponentWeights &value)
Definition: componentWeights.cpp:23
def __init__(self, engine, path=None)
Definition: backend/trt/loader.py:490
expr erfc(half arg)
Complementary error function.
Definition: ieee_half.h:2526
static std::string severityPrefix(Severity severity)
Definition: samples/common/logging.h:165
int mSampleCount
Definition: beamSearchPolicy.h:72
def preprocess(dynamic_graph)
Definition: samples/opensource/sampleUffFasterRCNN/config.py:33
int mKeepTopK
Definition: generateDetectionPlugin.h:96
int32_t anchorsScaleCount
Definition: NvInferPluginUtils.h:124
@ CENTER_SIZE
Use box centers and size.
def main()
Definition: main.py:30
def free(self)
Definition: infer.py:39
Definition: Polygraphy/polygraphy/logger/logger.py:58
def forward(self, input)
Definition: quant_instancenorm.py:44
def main()
Definition: multiproc.py:23
const char * message
Definition: checkMacrosPlugin.h:97
BatchedNMSPluginCreator()
Definition: batchedNMSPlugin.cpp:486
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: reorgPlugin.cpp:175
Generate an output tensor with specified mode.
Definition: NvInfer.h:4762
def __delitem__(self, index)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:86
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: debugUtil.cpp:33
expr fmod(half x, half y)
Remainder of division.
Definition: ieee_half.h:2243
def __init__(self, dir, name=None)
Definition: backend/tf/loader.py:162
n_frames_per_step
Definition: demo/Tacotron2/tacotron2/model.py:243
std::string mPluginNamespace
Definition: instanceNormalizationPlugin.h:99
Definition: blobNameToTensor.h:28
clip_value_max
Definition: clip.py:52
current_index
Definition: demo/BERT/helpers/calibrator.py:39
Definition: sampleOptions.h:169
Definition: sampleUffMaskRCNN.cpp:50
float variance[4]
Definition: NvInferPluginUtils.h:148
def test_equals_name_mismatch(self)
Definition: test_ir.py:116
~ProposalPlugin() override
Definition: proposalPlugin.cpp:148
def is_compressed(obj)
Definition: Polygraphy/polygraphy/util/misc.py:183
std::string mNamespace
Definition: proposalPlugin.h:145
const char * getPluginName() const override
Return the plugin name.
Definition: nmsPlugin.cpp:281
std::vector< size_t > topKMagnitudes(const std::vector< T > &v, const size_t k)
Definition: samples/common/common.h:443
expr pow(half base, half exp)
Power function.
Definition: ieee_half.h:2404
def _is_whitespace(char)
Definition: tokenization.py:392
def test_per_tensor_scale(self)
Definition: tensor_quant_test.py:141
def address(self)
Definition: cuda.py:103
virtual void setName(const char *name)=0
Set the name of a layer.
InferenceOptions inference
Definition: sampleOptions.h:191
virtual ~IPluginFactory()
Definition: NvInferRuntime.h:985
tensorboard_dir
Definition: backend/tf/loader.py:324
int32_t dlaCore
Specify the DLA core to run network on.
Definition: argsParser.h:39
Configuration Manager Class.
Definition: NvOnnxConfig.h:45
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: multilevelCropAndResizePlugin.cpp:212
static const float RPN_NMS_THRESHOLD
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:80
max_workspace_size
Definition: backend/tf/loader.py:226
Mask * mask
Definition: sampleUffMaskRCNN.cpp:68
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleCharRNN.cpp:146
def args
Definition: BERT/inference.py:83
bool useILoop
Definition: sampleCharRNN.cpp:133
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: pyramidROIAlignPlugin.cpp:58
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: regionPlugin.cpp:432
def get_input_metadata(self)
Definition: backend/trt_legacy.py:315
def extra_verbose(self, message, mode=LogMode.EACH)
Definition: Polygraphy/polygraphy/logger/logger.py:258
fp16
Definition: backend/tf/loader.py:227
~Encoder() override=default
int getC(const Dims &d)
Definition: samples/common/common.h:881
def test_multiple_profiles(self)
Definition: trt/test_runner.py:80
std::vector< SampleUniquePtr< nvinfer1::IHostMemory > > weightsMemory
Host weights memory holder.
Definition: sampleMLP.cpp:94
def set_input_metadata(self, input_metadata)
Definition: comparator/data_loader.py:265
def test_shape_output(self)
Definition: onnxrt/test_runner.py:45
def test_fold_constants_one_hop(self)
Definition: test_ir.py:726
output_attentions
Definition: quant_bert.py:57
float * getBatch() override
Definition: BatchStream.h:73
virtual void reset(int firstBatch)=0
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: multilevelCropAndResizePlugin.cpp:62
t0
Definition: Tacotron2/inference.py:173
n_layers
Definition: demo/Tacotron2/waveglow/model.py:93
const char * getFullTextFileName() const
Get the File Name of the Network Description as a Text File, including the weights.
Definition: sampleConfig.h:145
_engine_or_context
Definition: backend/trt/runner.py:50
def test_forward(self)
Definition: integration_test.py:62
void reset(int firstBatch) override
Definition: BatchStream.h:53
Definition: proposalPlugin.h:36
static expr divides(float x, float y)
Division implementation.
Definition: samples/common/half.h:1616
CropAndResizePluginCreator()
Definition: cropAndResizePlugin.cpp:243
@ kSEQUENCE
Elements correspond to different sequence values.
void setSoftmaxTree(const std::shared_ptr< softmaxTree > &softmaxTree)
Definition: regionPlugin.h:87
def register_callback(self, callback)
Definition: Polygraphy/polygraphy/logger/logger.py:150
TrtCudaEvent & operator=(const TrtCudaEvent &)=delete
std::shared_ptr< Attention > ptr
Definition: attention.h:35
def basic_cleaners(text)
Definition: cleaners.py:83
def float_function(fn)
Definition: amp_wrapper.py:26
Definition: backend/tf/loader.py:392
def test_track_amax(self)
Definition: calibrator_test.py:78
def __str__(self)
Definition: graph.py:466
def __init__(self, str path, List[Tensor] inputs, List[Tensor] outputs, List[Node] nodes, int opset)
Definition: onnx_models.py:35
reader of sequences of data
Definition: dataReader.h:31
std::string getFileType(const std::string &filepath)
Definition: samples/common/common.h:489
std::string mName
Definition: samples/common/common.h:220
static void help(std::ostream &out)
Definition: sampleOptions.cpp:1308
passed
Definition: compare.py:38
std::string ratingInputFile
Definition: sampleMovieLens.cpp:68
Definition: backend/trt_legacy.py:117
bias_ih
Definition: modules/quant_rnn.py:434
const char * getPluginType() const override
Return the plugin type.
Definition: gridAnchorPlugin.cpp:279
def forward(self, input)
Definition: quant_conv.py:160
virtual ~ISelectLayer()
Definition: NvInfer.h:4721
thrust::device_vector< float * > _d_output_ptrs
Definition: split.h:45
TrtCudaBuffer & operator=(TrtCudaBuffer &&rhs)
Definition: sampleDevice.h:229
GpuTimer(cudaStream_t stream)
Definition: samples/common/common.h:810
def identity_network()
Definition: trt/test_loader.py:48
static std::pair< const void *, size_t > getBlobProtoData(const trtcaffe::BlobProto &blobMsg, trtcaffe::Type type, std::vector< void * > &tmpAllocs)
Definition: caffeWeightFactory.cpp:279
void record(EventType e, StreamType s)
Definition: sampleInference.cpp:397
def __init__(self, tacotron2)
Definition: export_tacotron2_onnx.py:104
num_inferences
Definition: backend/tf/runner.py:54
Return type for specialized generic 2-argument half-precision functions.
Definition: ieee_half.h:393
int parseDLA(int argc, char **argv)
Definition: samples/common/common.h:602
bool useILoop
Definition: argsParser.h:89
IPluginV2Ext * deserializePlugin(const char *name, const void *data, size_t length) override
Called during deserialization of plugin layer.
Definition: detectionLayerPlugin.cpp:92
input_size
Definition: modules/quant_rnn.py:342
int mNumTokens
Definition: vocabulary.h:74
const char * getPluginVersion() const override
Return the plugin version.
Definition: reorgPlugin.cpp:195
def __init__(self, onnx_loader)
Definition: backend/trt_legacy.py:118
virtual nvinfer1::Dims getDims() const =0
c
Definition: 05_folding_constants/generate.py:31
static expr asin(float arg)
Arc sine implementation.
Definition: samples/common/half.h:1910
The SampleOnnxMNIST class implements the ONNX MNIST sample.
Definition: sampleOnnxMNIST.cpp:46
uint64_t getTopK() const
Definition: sampleConfig.h:277
std::mutex mRegistryLock
Definition: InferPlugin.cpp:142
void printMem(const char *where)
Definition: parserUtils.h:56
def set_tensor_name(tensor, prefix, name)
Definition: builder_varseqlen.py:111
def test_against_unquantized(self, verbose)
Definition: quant_rnn_test.py:380
def test_loader_explicit_precision(self)
Definition: trt/test_loader.py:113
tf_loader
Definition: backend/trt_legacy.py:50
Definition: demo/Tacotron2/tacotron2/model.py:48
bool isnormal(half arg)
Check if normal number.
Definition: ieee_half.h:2752
static half copysign(half x, half y)
Sign implementation.
Definition: samples/common/half.h:2321
bool verifyOutput(const samplesCommon::BufferManager &buffers)
Classifies digits and verify result.
Definition: sampleOnnxMNIST.cpp:271
nvinfer1::DimsHW preRefineSortedScoreDims
Definition: maskRCNNKernels.h:127
std::string engine
Definition: demo/BERT/infer_c/common.h:29
virtual nvinfer1::DataType getOutputDataType(int32_t index, const nvinfer1::DataType *inputTypes, int32_t nbInputs) const =0
Return the DataType of the plugin output at the requested index.
void * mMemory
Definition: binaryProtoBlob.h:61
~MultilevelCropAndResize() override=default
ComponentWeights()=default
Definition: tensor_quantizer.py:34
std::vector< nvinfer1::Weights > mGateBiasWeights
Definition: lstmDecoder.h:48
def __init__(self, **kwargs)
Definition: tools/pytorch-quantization/tests/fixtures/models.py:27
virtual void reportLayerTime(const char *layerName, float ms)
Layer time reporting callback.
Definition: samples/common/common.h:147
def __init__(self, c)
Definition: demo/Tacotron2/waveglow/model.py:39
nvinfer1::ITensor * inputBias
Definition: sampleCharRNN.cpp:252
def main(args=None)
Definition: mrcnn_to_trt_single.py:99
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: batchTilePlugin.cpp:120
def convert_ids_to_tokens(self, ids)
Definition: tokenization.py:208
void setFullTextFileName(const char *fullTextFilename)
Set the File Name of the Network Description as a Text File, including the weights.
Definition: sampleConfig.h:149
int32_t backgroundLabelId
Definition: NvInferPluginUtils.h:184
Descriptor for four-dimensional data.
Definition: NvInfer.h:290
void logError(const char *msg, const char *file, const char *fn, int line)
Definition: checkMacrosPlugin.cpp:94
Definition: backend/trt_legacy.py:1
int enqueue(const nvinfer1::PluginTensorDesc *inputDesc, const nvinfer1::PluginTensorDesc *outputDesc, const void *const *inputs, void *const *outputs, void *workspace, cudaStream_t stream) override
Execute the layer.
Definition: qkvToContextInt8InterleavedPlugin.cpp:221
virtual void setTextFileName(const char *textFileName)=0
Set the File Name of the Network Description as a Text File.
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: plugin.h:52
def log(self, message, severity, mode=LogMode.EACH, stack_depth=2)
Definition: Polygraphy/polygraphy/logger/logger.py:182
size_t getWorkspaceSize(int) const override
Definition: cropAndResizePlugin.h:58
def test_equals_inputs_mismatch(self)
Definition: test_ir.py:158
~ProposalLayer() override=default
bool infer()
Runs the TensorRT inference engine for this sample.
Definition: sampleMNISTAPI.cpp:262
void initializePlugin(void *logger, const char *libNamespace)
Definition: InferPlugin.cpp:152
n_remaining_channels
Definition: demo/Tacotron2/waveglow/model.py:183
name
Definition: onnx-graphsurgeon/tests/test_examples.py:35
std::vector< std::string > splitString(std::string str, char delimiter=',')
Definition: samples/common/common.h:861
Plugin class for user-implemented layers.
Definition: NvInferRuntime.h:113
BatchedNMSDynamicPluginCreator()
Definition: batchedNMSPlugin.cpp:491
logging_indent
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:101
def unmark_outputs(model, outputs)
Definition: onnx/util.py:97
bool processInput(const samplesCommon::BufferManager &buffers)
Reads the input and stores the result in a managed buffer.
Definition: sampleMLP.cpp:297
bool mShouldLog
Definition: samples/common/logging.h:178
std::string networkTensorsFileName
Definition: sampleINT8API.cpp:67
@ kDLA_LINEAR
DLA planar format.
static nvinfer1::PluginFieldCollection mFC
Definition: embLayerNormVarSeqlenPlugin.h:135
static half fmin(half x, half y)
Definition: samples/common/half.h:2585
def __init__(self, graph)
Definition: backend/tf/loader.py:28
int enqueue(int batch_size, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: generateDetectionPlugin.cpp:272
Definition: samples/common/common.h:781
calculates raw logits
Definition: projection.h:32
CUresult(* _cuLinkDestroy)(CUlinkState)
Definition: cudaDriverWrapper.h:74
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: batchTilePlugin.cpp:132
def test_find_in_dict(case)
Definition: test_misc.py:47
Definition: comparator/comparator.py:31
def finetune_model(model, data_loader)
Definition: classification_flow.py:412
def __setattr__(self, name, value)
Definition: graph.py:144
def test_exclude_outputs_with_layerwise(self)
Definition: onnx/test_loader.py:67
def disable_clip(self)
Definition: tensor_quantizer.py:157
nvinfer1::ITensor * inputWeights
Definition: sampleCharRNN.cpp:250
char shortName
Option name in short (single hyphen) form (i.e. -a, -b)
Definition: getOptions.h:36
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: priorBoxPlugin.cpp:294
std::string mCmdline
Definition: pluginLogging.h:304
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: qkvToContextInt8InterleavedPlugin.cpp:177
const char * getPluginVersion() const override
Return the plugin version.
Definition: sampleUffPluginV2Ext.cpp:653
nvinfer1::IPluginV2 * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: qkvToContextInt8InterleavedPlugin.cpp:345
std::string getInfo() override
get the textual description of the component
Definition: multiplicativeAlignment.cpp:76
engine
Definition: backend/trt/runner.py:84
constexpr uint32_t IIDX
Definition: qkvToContextInt8InterleavedPlugin.cpp:49
LogStreamConsumer(Severity reportableSeverity, Severity severity)
Creates a LogStreamConsumer which logs messages with level severity.
Definition: samples/common/logging.h:137
void setImageFileName(const char *imageFilename)
Definition: sampleConfig.h:237
@ kSPATIAL
Elements correspond to different spatial data.
float * copyToHost(const void *srcHostData, int count)
Definition: nvFasterRCNNPlugin.cpp:200
void attachToContext(cudnnContext *cudnn, cublasContext *cublas, nvinfer1::IGpuAllocator *allocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: groupNormalizationPlugin.cpp:92
DimsHW()
Construct an empty DimsHW object.
Definition: NvInfer.h:101
void reportAssertion(const char *msg, const char *file, int line)
Definition: checkMacrosPlugin.cpp:101
Class which manages logging of TensorRT tools and samples.
Definition: samples/common/logging.h:206
def test_will_not_give_up_on_first_cache_miss(self)
Definition: test_data_loader.py:118
def __init__(self, output_size, **kwargs)
Definition: quant_pooling.py:143
@ kINTERNAL_ERROR
A non-recoverable TensorRT error occurred.
gpu_memory_fraction
Definition: backend/tf/loader.py:369
bool build()
Function builds the network engine.
Definition: sampleUffFasterRCNN.cpp:163
const char * CELL_IN_BLOB_NAME
Definition: sampleCharRNN.cpp:89
int initialize() override
Initialize the layer for execution.
Definition: proposalPlugin.cpp:178
Definition: backend/base/runner.py:1
@ DATA_TYPE_INT4
Definition: fused_multihead_attention_common.h:29
T type
Definition: ieee_half.h:375
std::function< void(TrtCudaStream &)> EnqueueFunction
Definition: sampleInference.cpp:250
opt
Definition: backend/trt/loader.py:246
The PriorBox plugin layer generates the prior boxes of designated sizes and aspect ratios across all ...
Definition: NvInferPluginUtils.h:91
std::shared_ptr< std::ostream > mOutput
Definition: textWriter.h:49
def update(self, other)
Definition: comparator/struct.py:85
def _parse_cmudict(file)
Definition: cmudict.py:60
def forward(ctx, input, clip_value_min, clip_value_max)
Definition: functional.py:36
constexpr int defaultIterations
Definition: sampleOptions.h:46
float mTolerance
Definition: sampleConfig.h:59
static constexpr int32_t kSM_AMPERE
Definition: qkvToContextInt8InterleavedPlugin.h:31
TrtCudaStream mainStream
Definition: sampleInference.cpp:169
def create_network(builder, network)
Definition: Polygraphy/examples/api/05_using_tensorrt_network_api/example.py:34
void printProlog(int warmups, int timings, float warmupMs, float benchTimeMs, std::ostream &os)
Print benchmarking time and number of traces collected.
Definition: sampleReporting.cpp:79
enable< expr, T, U >::type operator*(T x, U y)
Multiply halfs.
Definition: ieee_half.h:2184
def __init__(self, config)
Definition: quant_bert.py:50
int width
Definition: batchPrepare.py:35
@ kCONCATENATION
Concatenation layer.
def get_mel_text_pair(self, audiopath_and_text)
Definition: tacotron2/data_function.py:45
virtual ~IErrorRecorder() noexcept
Definition: NvInferRuntimeCommon.h:1246
def get_batch(self, names)
Definition: demo/BERT/helpers/calibrator.py:58
std::shared_ptr< DumpTensorPlugin > ptr
Definition: debugUtil.h:41
max_seq_length
Definition: demo/BERT/helpers/calibrator.py:37
def __exit__(self, exc_type, exc_value, traceback)
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:51
Definition: backend/trt_legacy.py:35
int enqueue(int batch_size, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: pyramidROIAlignPlugin.cpp:185
Definition: sampleCharRNN.cpp:238
void configure(const Dims *, int32_t, const Dims *, int32_t, int32_t)
Derived classes should not implement this.
Definition: NvInferRuntime.h:279
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: generateDetectionPlugin.cpp:130
virtual std::vector< nvinfer1::Dims > getStateSizes()=0
get the sizes (vector of them) of the hidden state vectors
def __exit__(self, exc_type, exc_value, traceback)
Definition: Polygraphy/polygraphy/logger/logger.py:53
Definition: flattenConcat.h:108
def test_calibrator_caches_without_explicit_cache(self, identity_builder_network)
Definition: test_calibrator.py:114
use_int8_skipln
Definition: builder.py:101
_amax
Definition: tensor_quant.py:98
bool setTensorScales(const INetworkDefinition &network, float inScales=2.0f, float outScales=4.0f)
Definition: sampleEngines.cpp:254
std::vector< MultiEvent > mEvents
Definition: sampleInference.cpp:468
res_skip_layers
Definition: demo/Tacotron2/waveglow/model.py:96
const char * getPluginVersion() const override
Return the plugin version.
Definition: coordConvACPlugin.cpp:185
@ CORNER_SIZE
Use box centers and size.
unsigned char fused_multihead_attention_v2_fp16_256_64_kernel_sm80_cubin[]
Definition: fused_multihead_attention_v2_fp16_256_64_kernel.sm80.cpp:19
IBuilder * createInferBuilder(ILogger &logger)
Create an instance of an IBuilder class.
Definition: NvInfer.h:7440
def learn_amax(self)
Definition: tensor_quant.py:128
bool channelPivot
Definition: sampleReformatFreeIO.cpp:121
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: detectionLayerPlugin.cpp:282
float inStart
Definition: sampleReporting.h:94
int A
Definition: nvFasterRCNNPlugin.h:99
static expr plus(float x, float y)
Addition implementation.
Definition: samples/common/half.h:1589
const char * getPluginType() const override
Return the plugin type.
Definition: generateDetectionPlugin.cpp:165
Definition: fused_multihead_attention.h:28
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
def __init__(self, kernel_size, stride=None, padding=0, dilation=1, return_indices=False, ceil_mode=False, **kwargs)
Definition: quant_pooling.py:38
Logger & operator<<(T const &obj)
Definition: demo/BERT/infer_c/logging.h:38
def forward(self, input)
Definition: quant_instancenorm.py:76
def test_save_onnx(self)
Definition: onnx/test_loader.py:75
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: coordConvACPlugin.cpp:172
std::string mNamespace
Definition: batchedNMSPlugin.h:116
min
Definition: backend/trt/loader.py:245
def forward(self, input)
Definition: quant_linear.py:68
IPluginV2 * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: sampleUffPluginV2Ext.cpp:663
void setCalibrationFileName(const char *calibrationFilename)
Definition: sampleConfig.h:272
calibrator
Definition: backend/trt/loader.py:326
expr fdim(half x, half y)
Positive difference.
Definition: ieee_half.h:2309
def get_input_metadata(self)
Definition: backend/base/runner.py:123
nonlinearity
Definition: modules/quant_rnn.py:345
int outputSize
The output size.
Definition: sampleMLP.cpp:49
def LSTMCell(input, hidden, w_ih, w_hh, b_ih=None, b_hh=None, input_quantizer=None, weight_quantizer=None)
Definition: _functions/quant_rnn.py:46
def test_tf32(self, identity_builder_network, flag)
Definition: trt/test_loader.py:201
ModelFormat
Definition: sampleOptions.h:55
size_t mTokVocabSize
Definition: embLayerNormVarSeqlenPlugin.h:96
std::vector< std::string > dataDirs
Definition: argsParser.h:86
Describes a variation of execution of a layer. An algorithm is represented by IAlgorithmVariant and t...
Definition: NvInfer.h:6267
_ptr
Definition: cuda.py:125
void reset() override
Reset the reader position, the data reader is ready to read the data from th ebeginning again after t...
Definition: textReader.cpp:60
Definition: NvInferRuntime.h:356
virtual const char * file() const =0
source file in which the error occurred
std::unordered_map< std::string, ShapeRange > shapes
Definition: sampleOptions.h:123
def group_parameters(model, patterns_list, lrs=None, momentums=None, weight_decays=None)
Definition: helper.py:43
def _clean_text(self, text)
Definition: tokenization.py:316
std::vector< std::pair< int32_t, float > > itemProbPairVec
Definition: sampleMovieLensMPS.cpp:161
def str_from_graph(graph, mode)
Definition: tf/util.py:161
def test_input_fake_quant(self)
Definition: quant_pooling_test.py:228
half trunc(half arg)
Nearest integer not greater in magnitude than half value.
Definition: ieee_half.h:2565
TrtUniquePtr< nvcaffeparser1::ICaffeParser > caffeParser
Definition: sampleEngines.h:35
int mNumUnits
Definition: lstmDecoder.h:51
virtual void destroy() noexcept=0
Destroy this object.
@ kACTIVATION
Activation layer.
@ kDATATYPE
nvinfer1::DataType field type.
static bool isgreaterequal(half x, half y)
Comparison implementation.
Definition: samples/common/half.h:2420
int32_t numClasses
Definition: NvInferPluginUtils.h:250
def hasNames(features)
Definition: profiler.py:46
int mNbOutputChannels
Definition: samples/opensource/samplePlugin/fcPlugin.h:308
def __init__(self, cobj)
Definition: Polygraphy/polygraphy/util/misc.py:179
nvinfer1::Dims mInputDims
The dimensions of the input to the network.
Definition: sampleSSD.cpp:90
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: detectionLayerPlugin.cpp:231
InferenceTrace(int s, float es, float ee, float is, float ie, float cs, float ce, float os, float oe)
Definition: sampleReporting.h:71
bool processInput(const samplesCommon::BufferManager &buffers)
Reads the input and mean data, preprocesses, and stores the result in a managed buffer.
Definition: sampleFasterRCNN.cpp:259
std::string model
Definition: sampleOptions.h:77
SampleUffPluginV2Ext(const UffSampleParams ¶ms)
Definition: sampleUffPluginV2Ext.cpp:194
static const int MASK_POOL_SIZE
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:30
def test_add(self, field_names)
Definition: test_ir.py:294
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleSSD.cpp:94
Weights deserializeToDevice(const char *&hostBuffer, size_t count)
def test_import(self)
Definition: test_api.py:30
SimpleProfiler(const char *name, const std::vector< SimpleProfiler > &srcProfilers=std::vector< SimpleProfiler >())
Definition: samples/common/common.h:157
std::vector< float > mFileBatch
List of image files.
Definition: BatchStream.h:381
def __call__(self)
Definition: backend/onnx/loader.py:35
half & operator+=(float rhs)
Arithmetic assignment.
Definition: samples/common/half.h:1488
std::unique_ptr< FCPlugin > mPlugin
Definition: samples/opensource/samplePlugin/fcPlugin.h:381
CUresult cuLinkCreate(unsigned int numOptions, CUjit_option *options, void **optionValues, CUlinkState *stateOut) const
Definition: cudaDriverWrapper.cpp:101
EnqueueGraph(TrtCudaGraph &graph)
Definition: sampleInference.cpp:237
Definition: sampleUtils.h:215
Definition: quant_instancenorm.py:49
std::vector< void * > mDevicePointers
Definition: sampleUtils.h:453
Definition: calibrator_test.py:34
static const int TOP_DOWN_PYRAMID_SIZE
Definition: tlt_mrcnn_config.h:53
@ kFAILED_EXECUTION
An error occurred during execution that caused TensorRT to end prematurely, either an asynchronous er...
Definition: backend/onnx/loader.py:153
size_t getWorkspaceSize(int maxBatchSize) const override
Definition: regionPlugin.cpp:393
cudnnHandle_t mCudnn
Definition: samples/opensource/samplePlugin/fcPlugin.h:315
void writeNetworkTensorNames(const SampleUniquePtr< nvinfer1::INetworkDefinition > &network)
Write network tensor names to a file.
Definition: sampleINT8API.cpp:252
size_t mMaxBoxNum
Definition: proposalPlugin.h:118
IPluginV2Ext * createPlugin(const char *name, const PluginFieldCollection *fc) override
Return a plugin object.
Definition: multilevelProposeROIPlugin.cpp:69
static bool isnan(half arg)
Classification implementation.
Definition: ieee_half.h:1874
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: batchedNMSPlugin.cpp:99
def __call__(self)
Definition: backend/onnx/loader.py:189
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: proposalPlugin.cpp:289
static void reportTestStart(TestAtom &testAtom)
Report that a test has started.
Definition: samples/common/logging.h:324
unsigned int getElementSize(nvinfer1::DataType t)
Definition: samples/common/common.h:613
void configureWithFormat(const Dims *inputDims, int nbInputs, const Dims *outputDims, int nbOutputs, DataType type, PluginFormat format, int maxBatchSize) override
Definition: lReluPlugin.cpp:81
virtual bool canBroadcastInputAcrossBatch(int32_t inputIndex) const =0
Return true if plugin can use input that is broadcast across batch without replication.
def forward(self, input)
Definition: quant_pooling.py:98
def lstm_model()
Definition: onnx_models.py:114
constexpr int32_t EnumMax< PoolingType >()
Maximum number of elements in PoolingType enum.
Definition: NvInfer.h:1715
@ kEXPLICIT_BATCH
Dynamic shape support requires that the kEXPLICIT_BATCH flag is set.
SplitPlugin(int axis, std::vector< int > output_lengths)
Definition: split.h:75
LogStreamConsumerBase(std::ostream &stream, const std::string &prefix, bool shouldLog)
Definition: pluginLogging.h:110
def _is_control(char)
Definition: tokenization.py:404
inputs
Definition: tensor.py:198
def test_postprocess(self)
Definition: test_comparator.py:86
@ kPROD
Product of the two operands.
stream
Definition: BERT/inference.py:151
def str_from_module_info(module, name=None)
Definition: Polygraphy/polygraphy/util/misc.py:395
int mEndSequenceId
Definition: beamSearchPolicy.h:66
name
Definition: tensor.py:56
def activate_impl(self)
Definition: backend/cntk/runner.py:31
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: reorgPlugin.cpp:126
def activate_impl(self)
Definition: backend/onnx/runner.py:38
def add_binding_to_metadata(engine, binding, metadata)
Definition: trt/util.py:365
const int kDIGITS
Definition: sampleUffMNIST.cpp:104
indent
Definition: Polygraphy/polygraphy/logger/logger.py:47
friend std::ostream & operator<<(std::ostream &out, const SimpleProfiler &value)
Definition: samples/common/common.h:178
def __call__(self)
Definition: backend/trt_legacy.py:159
const char * getPluginName() const override
Return the plugin name.
Definition: multilevelCropAndResizePlugin.cpp:47
letter
Definition: onnx-graphsurgeon/onnx_graphsurgeon/logger/logger.py:105
virtual ~IPluginFactory()
Definition: NvUffParser.h:121
infer
Definition: onnx-graphsurgeon/tests/test_examples.py:36
bool buildPredictionEngine(const SampleUniquePtr< nvinfer1::IBuilder > &builder)
Builds an engine for prediction (mPredictionEngine).
Definition: sampleDynamicReshape.cpp:208
Definition: samples/common/common.h:139
def test_calibrator_generator_data(self, identity_builder_network)
Definition: test_calibrator.py:52
bool supportsFormatCombination(int pos, const nvinfer1::PluginTensorDesc *inOut, int nbInputs, int nbOutputs) override
Definition: skipLayerNormInt8InterleavedPlugin.cpp:119
def test_tensors_check_duplicates(self)
Definition: test_ir.py:587
std::string imageFileName
Definition: sampleINT8API.cpp:652
int mOutputTokenCount
Definition: benchmarkWriter.h:50
virtual ~ILoop()
Definition: NvInfer.h:4712
def load_config(config)
Definition: Polygraphy/examples/api/03_interoperating_with_tensorrt/example.py:48
bias_ih
Definition: modules/quant_rnn.py:349
static std::string genCmdlineString(int argc, char const *const *argv)
generate a command line string from the given (argc, argv) values
Definition: pluginLogging.h:448
virtual ~IElementWiseLayer()
Definition: NvInfer.h:2668
Definition: samples/common/common.h:840
Definition: sampleCharRNN.cpp:114
~BeamSearchPolicy() override=default
int mMaxBatchSize
Definition: detectionLayerPlugin.h:99
nvinfer1::DimsHW argMaxBboxDims
Definition: maskRCNNKernels.h:130
def test_i(self)
Definition: test_ir.py:80
def __call__(self)
Definition: backend/trt/loader.py:75
std::vector< Ray > mBeamSearchTable
Definition: beamSearchPolicy.h:71
void reduceVerbosity()
Decrease verbosity Level.
Definition: sampleConfig.h:121
def test_no_optimize(self)
Definition: onnx/test_loader.py:42
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: proposalPlugin.cpp:200
IPluginV2Ext * clone() const override
Clone the plugin object.
Definition: nvFasterRCNNPlugin.cpp:236
void putOutput()
Definition: samples/common/logging.h:72
DataType mDataType
Definition: sampleUffPluginV2Ext.cpp:636
def get_lists(self, field_names)
Definition: test_ir.py:226
void serialize(void *buffer) const override
Serialize the layer.
Definition: flattenConcat.cpp:167
int nbCalBatches
The number of batches for calibration.
Definition: sampleSSD.cpp:51
const char * getPluginName() const override
Return the plugin name.
Definition: resizeNearestPlugin.cpp:47
virtual void attachToContext(cudnnContext *, cublasContext *, IGpuAllocator *)
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: NvInferRuntimeCommon.h:667
@ kINDEX
Elements correspond to different batch index.
def __init__(self, n_mel_channels, postnet_embedding_dim, postnet_kernel_size, postnet_n_convolutions)
Definition: demo/Tacotron2/tacotron2/model.py:127
Inference iteration and streams management.
Definition: sampleInference.cpp:281
virtual nvinfer1::ILayer * addLSTMLayers(SampleUniquePtr< nvinfer1::INetworkDefinition > &network)=0
Add inputs to the TensorRT network and configure LSTM layers using network definition API.
n_frames_per_step
Definition: demo/Tacotron2/tacotron2/model.py:592
Definition: backend/tf/loader.py:1
void cudaSleep(cudaStream_t stream, cudaError_t status, void *sleep)
Definition: sampleDevice.h:43
void destroy() override
Destroy the plugin object.
Definition: nmsPlugin.cpp:160
void loadLibrary(const std::string &path)
Definition: samples/common/common.h:896
_sess
Definition: backend/onnxrt/runner.py:36
int mNbClasses
Definition: generateDetectionPlugin.h:95
Slices an input tensor into an output tensor based on the offset and strides.
Definition: NvInfer.h:3922
@ kPARAMETRIC_RELU
Parametric ReLU layer.
Definition: backend/trt_legacy.py:174
def flatten_parameters(self)
Definition: modules/quant_rnn.py:103
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: split.h:132
int initialize() override
Initialize the layer for execution.
Definition: generateDetectionPlugin.cpp:135
std::vector< size_t > mPossibleMatchesByOrder
Definition: bleuScoreWriter.h:59
TopKOperation
Enumerates the operations that may be performed by a TopK layer.
Definition: NvInfer.h:4055
encodes input sentences into output states using LSTM
Definition: lstmEncoder.h:31
float value_
Internal expression value stored in single-precision.
Definition: ieee_half.h:365
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace of the plugin creator based on the plugin library it belongs to.
Definition: groupNormalizationPlugin.cpp:280
int mNbInputChannels
Definition: samples/opensource/samplePlugin/fcPlugin.h:308
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: specialSlicePlugin.h:51
virtual IPluginV2 * createPlugin(const char *name, const PluginFieldCollection *fc)=0
Return a plugin object.
path
Definition: backend/tf/loader.py:323
Definition: softmaxLikelihood.h:33
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: batchedNMSPlugin.cpp:75
Convenience object used to initialize LogStreamConsumerBuffer before std::ostream in LogStreamConsume...
Definition: samples/common/logging.h:111
Definition: BERT/inference.py:1
void detachFromContext() override
Detach the plugin object from its execution context.
Definition: detectionLayerPlugin.cpp:313
static PluginFieldCollection mFC
Definition: batchedNMSPlugin.h:132
const TFusedMHAKernelList * getXMMAKernels(const typename TFusedMHAKernelList::KernelMeta *pKernelList, unsigned int nbKernels, Data_type type, unsigned int sm)
Definition: fused_multihead_attention.h:295
const char * getPluginVersion() const override
Return the plugin version.
Definition: multilevelCropAndResizePlugin.cpp:135
const char * getPluginName() const override
Return the plugin name.
Definition: regionPlugin.cpp:479
errorStack mErrorStack
Definition: ErrorRecorder.h:132
IPluginV2DynamicExt * deserializePlugin(const char *name, const void *serialData, size_t serialLength) override
Called during deserialization of plugin layer.
Definition: instanceNormalizationPlugin.cpp:399
def ultra_verbose(self, message, mode=LogMode.EACH)
Definition: Polygraphy/polygraphy/logger/logger.py:250
nvinfer1::DimsHW preRefineScoreDims
Definition: maskRCNNKernels.h:126
std::chrono::high_resolution_clock::time_point mStartTS
Definition: benchmarkWriter.h:51
std::string mPluginNamespace
Definition: normalizePlugin.h:104
void destroy() override
Destroy the plugin object.
Definition: gridAnchorPlugin.cpp:339
int initialize() override
Initialize the layer for execution.
Definition: nmsPlugin.cpp:73
std::vector< std::unique_ptr< Bindings > > bindings
Definition: sampleInference.h:38
def copyright_scan(directory, depth, args, exclude_dirs=[])
Definition: copyright-scan.py:152
def test_export_node(self)
Definition: test_exporters.py:80
enable< expr, T, U >::type operator/(T x, U y)
Divide halfs.
Definition: ieee_half.h:2190
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: pyramidROIAlignPlugin.cpp:88
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: cropAndResizePlugin.cpp:142
const char * function
Definition: checkMacrosPlugin.h:94
hidden_size
Definition: modules/quant_rnn.py:343
TensorLocation
The location for tensor data storage, device or host.
Definition: NvInferRuntimeCommon.h:1015
Definition: quant_rnn_test.py:255
RPROIParams params
Definition: nvFasterRCNNPlugin.h:98
void allocate(size_t size)
Definition: sampleDevice.h:249
const char * getPluginVersion() const override
Return the plugin version.
Definition: nvFasterRCNNPlugin.cpp:224
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: normalizePlugin.cpp:221
FCPlugin(const void *data, size_t length)
Definition: samples/opensource/samplePlugin/fcPlugin.h:62
std::vector< OutputParams > outParamsVec
Definition: sampleMovieLens.cpp:75
static half modf(half arg, half *iptr)
Decompression implementation.
Definition: samples/common/half.h:2167
bool isPlugin(const char *name) override
A user implemented function that determines if a layer configuration is provided by an IPlugin.
Definition: samples/opensource/samplePlugin/fcPlugin.h:325
@ kFULLY_CONNECTED
Fully connected layer.
def test_can_copy_inputs_from_other_node(self)
Definition: test_ir.py:67
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: multilevelCropAndResizePlugin.cpp:254
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleFasterRCNN.cpp:58
void attachToContext(cudnnContext *cudnn, cublasContext *cublas, nvinfer1::IGpuAllocator *allocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: instanceNormalizationPlugin.cpp:280
std::string mListFile
File name of the list of image names.
Definition: BatchStream.h:386
const char * NORMALIZE_PLUGIN_VERSION
Definition: normalizePlugin.cpp:31
static const std::string mUnkStr
Definition: vocabulary.h:69
nvinfer1::DimsHW argMaxScoreDims
Definition: maskRCNNKernels.h:164
def chpt_to_dict_arrays()
Definition: chptToBin.py:54
Fields that a plugin might see for an input or output.
Definition: NvInferRuntimeCommon.h:367
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool *inputIsBroadcasted, int nbInputs) const override
Definition: multilevelCropAndResizePlugin.cpp:261
def test_simple_run(self)
Definition: tensor_quant_test.py:40
float expectedPredictedMaxRatingItemProb
Definition: sampleMovieLens.cpp:53
static half_float::half lowest()
Smallest finite value.
Definition: samples/common/half.h:4222
int getBatchesRead() const override
Definition: BatchStream.h:269
virtual void terminate()=0
Release resources acquired during plugin layer initialization.
std::vector< nvinfer1::PluginField > parseRPROIParam(const trtcaffe::LayerParameter &msg, CaffeWeightFactory &weightFactory, BlobNameToTensor &tensors)
Definition: caffeParser.cpp:245
bool build()
Function builds the network engine.
Definition: sampleSSD.cpp:122
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: reorgPlugin.cpp:63
bool checkForNans(const void *values, int count, const std::string &layerName)
Definition: caffeWeightFactory.cpp:381
@ kFORGET
Forget gate (f).
Definition: sampleUffMaskRCNN.cpp:57
def get_mask_from_lengths(lengths)
Definition: demo/Tacotron2/common/utils.py:23
uint16 float2half_impl(float value, true_type)
Convert IEEE single-precision to half-precision.
Definition: ieee_half.h:454
Descriptor for data with one index dimension, one channel dimension and two spatial dimensions.
def transformer_layer_opt(prefix, config, init_dict, network, input_tensor, mask_idx, cu_seqlens, max_seqlen)
Definition: builder_varseqlen.py:208
Dims getOutputDimensions(int index, const Dims *inputs, int nbInputDims) override
Definition: nvFasterRCNNPlugin.cpp:127
nvinfer1::ILogger & getTRTLogger()
Forward-compatible method for retrieving the nvinfer::ILogger associated with this Logger.
Definition: samples/common/logging.h:233
void writeCalibrationCache(const void *cache, size_t length) override
Definition: EntropyCalibrator.h:126
bool teardown()
Cleans up any state created in the sample class.
Definition: sampleMNISTAPI.cpp:378
unsigned char fused_multihead_attention_v2_fp16_384_64_kernel_sm75_cubin[]
Definition: fused_multihead_attention_v2_fp16_384_64_kernel.sm75.cpp:19
def main(_)
Definition: chptToBin.py:358
Definition: BatchStream.h:27
virtual IExecutionContext * createExecutionContext() noexcept=0
Create an execution context.
const char * SPECIALSLICE_PLUGIN_NAME
Definition: specialSlicePlugin.cpp:29
_config
Definition: backend/trt/loader.py:407
def __init__(self, n_frames_per_step)
Definition: tacotron2/data_function.py:86
CUresult cuLaunchKernel(CUfunction f, unsigned int gridDimX, unsigned int gridDimY, unsigned int gridDimZ, unsigned int blockDimX, unsigned int blockDimY, unsigned int blockDimZ, unsigned int sharedMemBytes, CUstream hStream, void **kernelParams, void **extra) const
Definition: cudaDriverWrapper.cpp:132
def save_layer_weights(data, list_keys, dims, footer_string, file_name)
Definition: chptToBin.py:331
PPM< uint8_t > resizeMask(const BBoxInfo &box, const float mask_threshold)
Definition: sampleUffMaskRCNN.cpp:190
_model
Definition: backend/onnx/runner.py:35
void reportAlgorithms(const nvinfer1::IAlgorithmContext *const *algoContexts, const nvinfer1::IAlgorithm *const *algoChoices, int32_t nbAlgorithms) noexcept override
called by TensorRT to report choices it made.
Definition: sampleAlgorithmSelector.cpp:76
def reset_parameters(self)
Definition: modules/quant_rnn.py:147
def get_collate_function(model_name, n_frames_per_step)
Definition: data_functions.py:25
def parse_waveglow_args(parent, add_help=False)
Definition: waveglow/arg_parser.py:19
def amaxs(self)
Definition: max.py:46
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: multilevelProposeROIPlugin.cpp:429
bool profile
Definition: sampleUffFasterRCNN.cpp:83
detail::uint16 data_
Internal binary representation.
Definition: ieee_half.h:1197
std::vector< int > _output_lengths
Definition: split.h:41
_data_ptrs
Definition: modules/quant_rnn.py:111
int outputClassSize
Definition: sampleUffFasterRCNN.cpp:71
const char * getPluginVersion() const override
Return the plugin version.
Definition: lReluPlugin.cpp:115
Definition: quant_conv.py:41
_narrow_range
Definition: tensor_quant.py:105
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: qkvToContextInt8InterleavedPlugin.cpp:187
def toposort_multi_tier_input_graph()
Definition: test_ir.py:430
def str_from_onnx_graph(graph, mode, tensors, indent_level=0)
Definition: onnx/util.py:194
static expr pow(float base, float exp)
Power implementation.
Definition: samples/common/half.h:1878
std::string getInfo() override
get the textual description of the component
Definition: lstmDecoder.cpp:142
def get_tensor_metadata(tensors)
Definition: onnx/util.py:150
Definition: tensor_quant_test.py:269
int getBatchSize() const override
Get the batch size used for calibration batches.
Definition: sampleEngines.cpp:173
int enqueue(int batchSize, const void *const *inputs, void **outputs, void *workspace, cudaStream_t stream) override
Definition: regionPlugin.cpp:246
unsigned int fused_multihead_attention_v2_fp16_64_64_kernel_sm75_cubin_len
Definition: fused_multihead_attention_v2_fp16_64_64_kernel.sm75.cpp:1620
Definition: tensor.py:186
int getNbOutputs() const override
Get the number of outputs from the layer.
Definition: sampleUffPluginV2Ext.cpp:386
The SampleINT8APIArgs structures groups the additional arguments required by the INT8 API sample.
Definition: sampleINT8API.cpp:647
nvinfer1::ILayer * parseReLU(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &, BlobNameToTensor &tensors)
Definition: parseReLU.cpp:23
bool writeNetworkTensors
Definition: sampleINT8API.cpp:58
static const Dims2 MODEL_DETECTION_SHAPE
Definition: tlt_mrcnn_config.h:174
virtual ITensor * getInput(int32_t index) const =0
Get the input tensor specified by the given index.
const char * getPluginVersion() const override
Return the plugin version.
Definition: proposalPlugin.cpp:339
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: regionPlugin.cpp:264
def get_final_text(pred_text, orig_text, do_lower_case)
Definition: data_processing.py:224
def axis(self)
Definition: tensor_quantizer.py:139
std::vector< int32_t > allItems
Definition: sampleMovieLens.cpp:54
max_query_length
Definition: demo/BERT/helpers/calibrator.py:43
unsigned int elementSize(nvinfer1::DataType t)
Definition: parserUtils.h:74
def reshape(self, data, shape)
Definition: 09_shape_operations_with_the_layer_api/generate.py:36
uint32_t scale_bmm1
Definition: fused_multihead_attention_v2.h:58
def test_contains(self, run_results)
Definition: comparator/test_struct.py:87
__host__ __device__ __forceinline__ unsigned int umulhi(unsigned int x, unsigned int y)
Definition: reducedMathPlugin.h:35
float computeStart
Definition: sampleReporting.h:96
half & operator-=(float rhs)
Arithmetic assignment.
Definition: samples/common/half.h:1497
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: detectionLayerPlugin.cpp:288
Definition: onnx_models.py:34
int _z_stride
Definition: split.h:43
CalibrationAlgoType getAlgorithm() override
Signal that this is the MinMax Calibrator.
Definition: NvInfer.h:6099
virtual ~IPluginFactory()
Definition: NvCaffeParser.h:108
virtual bool refitCudaEngine()=0
Updates associated engine.
DataType getOutputDataType(int index, const DataType *inputTypes, int nbInputs) const override
Definition: sampleUffPluginV2Ext.cpp:530
bool mRNNKind
Definition: lstmDecoder.h:49
CudnnError(const char *fl, const char *fn, int ln, int stat, const char *msg=nullptr)
Definition: checkMacrosPlugin.h:113
Definition: checkMacrosPlugin.h:101
nvinfer1::ILayer * parseReduction(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &weightFactory, BlobNameToTensor &tensors)
Definition: parseReduction.cpp:23
def critical(self, message)
Definition: Polygraphy/polygraphy/logger/logger.py:287
_model
Definition: backend/onnx/loader.py:162
def parse_model_args(model_name, parser, add_help=False)
Definition: demo/Tacotron2/models.py:26
LogStreamConsumer LOG_WARN(const Logger &logger)
produces a LogStreamConsumer object that can be used to log messages of severity kWARNING
Definition: pluginLogging.h:497
Definition: quant_linear_test.py:44
nvinfer1::DimsHW argMaxLabelDims
Definition: maskRCNNKernels.h:166
IPluginV2Ext * deserializePlugin(const char *name, const void *data, size_t length) override
Called during deserialization of plugin layer.
Definition: multilevelProposeROIPlugin.cpp:106
def load_calib_amax(self, *args, **kwargs)
Definition: tensor_quantizer.py:218
virtual IPluginV2Ext * clone() const =0
Clone the plugin object.
int sync() override
Definition: checkMacrosPlugin.cpp:31
def quantize_by_range(x, num_bits)
Definition: tools/pytorch-quantization/tests/utils.py:26
int sleep
Definition: sampleOptions.h:152
~PluginCreatorRegistry()
Definition: InferPlugin.cpp:127
void addToModel(nvinfer1::INetworkDefinition *network, nvinfer1::ITensor *input, nvinfer1::ITensor **outputLogits) override
add raw logits to the network
Definition: slpProjection.cpp:41
TrtCudaGraph & mGraph
Definition: sampleInference.cpp:247
SampleUffFasterRcnnParams mParams
The parameters for the sample.
Definition: sampleUffFasterRCNN.cpp:121
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: reorgPlugin.cpp:72
def infer(self, spect, sigma=1.0)
Definition: demo/Tacotron2/waveglow/model.py:231
std::vector< std::string > outputSentences
Definition: sampleCharRNN.cpp:132
Definition: regionPlugin.cpp:46
old_severity
Definition: Polygraphy/polygraphy/logger/logger.py:61
TrtCudaEvent & getEvent(EventType t)
Definition: sampleInference.cpp:392
virtual void writeCalibrationCache(const void *, size_t) override
Definition: sampleEngines.cpp:180
Linear logits calculation.
Definition: slpProjection.h:36
def load_engine(engine_filepath, trt_logger)
Definition: trt_utils.py:46
Normalize(const Weights *weights, int nbWeights, bool acrossSpatial, bool channelShared, float eps)
Definition: normalizePlugin.cpp:38
~SamplePlugin()
Definition: samplePlugin.cpp:66
def question_features(tokens, question)
Definition: inference_varseqlen.py:113
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: instanceNormalizationPlugin.cpp:272
def no_publish()
Definition: onnx-graphsurgeon/setup.py:21
int w
Definition: samples/common/common.h:655
void serialize(void *buffer) const override
Serialize the layer.
Definition: instanceNormalizationPlugin.cpp:223
uint16 round_half_impl(uint16 value)
Round half-precision number to nearest integer value.
Definition: ieee_half.h:1008
const char * getPluginNamespace() const override
Return the namespace of the plugin object.
Definition: gridAnchorPlugin.cpp:295
def get_values(tensor)
Definition: onnx/util.py:144
void * o_ptr
Definition: fused_multihead_attention.h:55
void setPluginNamespace(const char *libNamespace) override
Set the namespace that this plugin object belongs to.
Definition: batchedNMSPlugin.cpp:409
@ kMATRIX_MULTIPLY
Matrix multiply layer.
size_t mNumboxes
Definition: cropAndResizePlugin.h:101
void launch_small(cudaStream_t stream, const int ld, const int total, const int8_t *input, const int8_t *skip, const half *beta, const half *gamma, int8_t *output, const float dqScaleIn, const float dqScaleSkip, const float qScale)
Definition: sampleUffFasterRCNN.cpp:93
bool islessgreater(half x, half y)
Comarison for less or greater.
Definition: ieee_half.h:2817
def test_can_name_runner(self)
Definition: trt/test_runner.py:36
virtual size_t getSerializationSize() const =0
Find the size of the serialization buffer required.
TrtCudaEvent(bool blocking=true)
Definition: sampleDevice.h:110
std::string mPluginNamespace
Definition: nvFasterRCNNPlugin.h:95
Definition: sampleDevice.h:273
BenchmarkWriter()
Definition: benchmarkWriter.cpp:24
static TestAtom defineTest(const std::string &name, const std::string &cmdline)
Define a test for logging.
Definition: pluginLogging.h:318
int mROICount
Definition: multilevelCropAndResizePlugin.h:97
def default_value(value, default)
Definition: Polygraphy/polygraphy/util/misc.py:101
int open_rw()
Definition: sampleMovieLensMPS.cpp:136
int sequenceLength
Definition: demo/BERT/infer_c/common.h:31
The SampleSSDParams structure groups the additional parameters required by the SSD sample.
Definition: sampleSSD.cpp:47
PinnedHostBuffer(size_t elementCount)
Definition: pinnedHostBuffer.h:37
NMSParameters params
Definition: batchedNMSPlugin.h:133
def all_tensor_names(model)
Definition: onnx/util.py:56
def build_two_layer_graph_multiple_io()
Definition: test_ir.py:358
Definition: tensor_quantizer.py:1
Verbosity mVerbosity
Definition: sampleConfig.h:53
num_attention_heads
Definition: builder.py:92
bool int8
Allow runnning the network in Int8 mode.
Definition: argsParser.h:40
The SampleUffMNIST class implements the UffMNIST sample.
Definition: sampleUffMNIST.cpp:53
def test_fake_quant_per_channel(self)
Definition: quant_linear_test.py:136
DataType mPrecision
Definition: batchedNMSPlugin.h:118
Plugin factory used to configure plugins with added support for TRT versioning.
Definition: NvUffParser.h:129
def str_from_onnx(model, mode="full")
Definition: onnx/util.py:168
Definition: multilevelProposeROIPlugin.h:35
def test_no_quant_input_hidden(self, verbose)
Definition: quant_rnn_test.py:85
AlgorithmCacheWriter(const std::string &cacheFileName)
Definition: sampleAlgorithmSelector.cpp:108
nvinfer1::ILayer * parseScale(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &weightFactory, BlobNameToTensor &tensors)
Definition: parseScale.cpp:23
decoder_rnn
Definition: export_tacotron2_onnx.py:123
int32_t keepTopK
The maximum number of detection post-NMS.
Definition: sampleUffSSD.cpp:55
Definition: quant_conv_transposed_test.py:238
QKVToContextInterleavedPluginCreator()
Definition: qkvToContextInt8InterleavedPlugin.cpp:276
Definition: resizeNearestPlugin.h:96
static expr sinh(float arg)
Hyperbolic sine implementation.
Definition: samples/common/half.h:1943
runner_name
Definition: comparator/struct.py:46
def forward(self, input, output_size=None)
Definition: quant_conv.py:378
constexpr int32_t EnumMax< ReduceOperation >()
Maximum number of elements in ReduceOperation enum.
Definition: NvInfer.h:3567
const std::string FCB_NAME
Definition: sampleCharRNN.cpp:78
int ilogb(half arg)
Extract exponent.
Definition: ieee_half.h:2669
def test_equals_num_inputs_mismatch(self)
Definition: test_ir.py:151
T half2int_impl(uint16 value)
Convert half-precision floating point to integer.
Definition: ieee_half.h:957
bool supportsFormat(DataType type, PluginFormat format) const override
Check format support.
Definition: proposalLayerPlugin.cpp:160
int W
Definition: reorgPlugin.h:87
static half rint(half arg)
Nearest integer implementation.
Definition: samples/common/half.h:2118
std::vector< std::vector< int > > mCandidates
Definition: beamSearchPolicy.h:76
samplesCommon::BBox box
Definition: sampleUffMaskRCNN.cpp:64
int32_t d[MAX_DIMS]
The extent of each dimension.
Definition: NvInferRuntimeCommon.h:199
def RNN(*args, **kwargs)
Definition: _functions/quant_rnn.py:279
void readPPMFile(const std::string &filename, PPM< uint8_t > &ppm)
Definition: sampleUffMaskRCNN.cpp:79
vocab
Definition: tokenization.py:164
void fill(int binding, const std::string &fileName)
Definition: sampleUtils.h:361
Definition: backend/cntk/runner.py:25
SampleGoogleNet(const samplesCommon::CaffeSampleParams ¶ms)
Definition: sampleGoogleNet.cpp:52
~TrtCudaBuffer()
Definition: sampleDevice.h:239
def pop_quant_desc_in_kwargs(quant_cls, input_only=False, **kwargs)
Definition: _utils.py:139
Class used for parsing models described using the UFF format.
Definition: NvUffParser.h:152
nvinfer1::DimsHW sortClassLabelDims
Definition: maskRCNNKernels.h:101
def load_and_setup_model(model_name, parser, checkpoint, amp_run, to_cuda=True)
Definition: test_infer_trt.py:72
const char * getPluginType() const override
Return the plugin type.
Definition: nvFasterRCNNPlugin.cpp:219
int8
Definition: backend/trt/loader.py:324
_if_calib
Definition: tensor_quantizer.py:86
int getBufferSize()
Definition: sampleReformatFreeIO.cpp:127
plugins
Definition: backend/trt/loader.py:49
Definition: histogram.py:30
Graph import_onnx("onnx.ModelProto" onnx_model)
Definition: onnx_importer.py:225
def test_can_add(self)
Definition: trt/test_loader.py:161
bool constructNetwork(SampleUniquePtr< nvinfer1::IBuilder > &builder, SampleUniquePtr< nvinfer1::INetworkDefinition > &network, SampleUniquePtr< nvinfer1::IBuilderConfig > &config, SampleUniquePtr< nvonnxparser::IParser > &parser)
Parses an ONNX model for MNIST and creates a TensorRT network.
Definition: sampleOnnxMNIST.cpp:163
nvinfer1::TacticSources disabledTactics
Definition: sampleOptions.h:128
Severity mSeverity
Definition: pluginLogging.h:181
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: batchTilePlugin.cpp:122
def test_one_tensor(self, verbose)
Definition: calibrator_test.py:251
def dynamic_range_compression(x, C=1, clip_val=1e-5)
Definition: audio_processing.py:94
const char * getPluginName() const override
Return the plugin name.
Definition: qkvToContextInt8InterleavedPlugin.cpp:282
std::vector< float > mAnchorBoxesHost
Definition: proposalLayerPlugin.h:107
nvinfer1::ILayer * parseBatchNormalization(nvinfer1::INetworkDefinition &network, const trtcaffe::LayerParameter &msg, CaffeWeightFactory &weightFactory, BlobNameToTensor &tensors)
Definition: parseBatchNorm.cpp:80
T type
Definition: ieee_half.h:384
virtual void addAttentionKeys(nvinfer1::INetworkDefinition *network, nvinfer1::ITensor *memoryStates, nvinfer1::ITensor **attentionKeys)=0
add attention keys calculation (from source memory states) to the network
def test_mark_custom_outputs(self, identity_identity_network)
Definition: trt/test_loader.py:130
def convert_example_to_features(doc_tokens, question_text, tokenizer, max_seq_length, doc_stride, max_query_length)
Definition: data_processing.py:86
path
Definition: backend/tf/loader.py:147
static int ilogb(half arg)
Exponent implementation.
Definition: samples/common/half.h:2236
Layer that represents a TopK reduction.
Definition: NvInfer.h:4075
def dtoh(self, dst, src, nbytes, stream=None)
Definition: cuda.py:76
const char * MULTILEVELPROPOSEROI_PLUGIN_NAME
Definition: multilevelProposeROIPlugin.cpp:35
def check(self, status)
Definition: cuda.py:39
Definition: argsParser.h:29
def quant_np(x, amax, num_bits=8, fake=False, narrow_range=True)
Definition: tools/pytorch-quantization/tests/utils.py:118
def test_cli_inspect_examples(example)
Definition: Polygraphy/tests/test_examples.py:135
int embSkipLayerNormVarSeqlen(cudaStream_t stream, int ld, int B, int S, const uint32_t *cuSeqlens, const int *inputIds, const int *token_ids, const T *beta, const T *gamma, const T *wordEmb, const T *posEmb, const T *tokEmb, T *output)
def test_can_copy_outputs_from_other_node(self)
Definition: test_ir.py:184
static const Dims2 MODEL_DETECTION_SHAPE
Definition: plugin/proposalLayerPlugin/mrcnn_config.h:174
half frexp(half arg, int *exp)
Decompress floating point number.
Definition: ieee_half.h:2627
std::string weightsFileName
Filename of trained weights file of a network.
Definition: argsParser.h:54
@ DATA_TYPE_FP32
Definition: fused_multihead_attention_common.h:28
size_t getSerializationSize() const override
Find the size of the serialization buffer required.
Definition: sampleUffPluginV2Ext.cpp:459
CalibrationAlgoType
enum CalibrationAlgoType
Definition: NvInfer.h:5972
virtual void destroy()
Destroy IOnnxConfig object.
Definition: parserOnnxConfig.h:146
F type
Definition: samples/common/half.h:329
IBinaryProtoBlob * parseBinaryProto(const char *fileName) override
Parse and extract data stored in binaryproto file.
Definition: caffeParser.cpp:654
T ymin
Definition: bboxUtils.h:28
virtual int32_t initialize()=0
Initialize the layer for execution.
def get_tensor_metadata(tensors)
Definition: tf/util.py:71
@ kSAFE_GPU
Safety restricted capability, TensorRT flow that can only run on GPU devices via TensorRT nvinfer1::s...
size_t mSize
Definition: maskRCNNKernels.h:65
def forward(self, input)
Definition: quant_pooling.py:45
MirroredBuffer buffer
Definition: sampleUtils.h:218
bias
Definition: modules/quant_rnn.py:344
def activate(self)
Definition: backend/base/runner.py:83
def get_output_metadata_from_engine(engine, start_binding, end_binding)
Definition: trt/util.py:381
std::unique_ptr< T, samplesCommon::InferDeleter > SampleUniquePtr
Definition: sampleMovieLens.cpp:86
Definition: nvFasterRCNNPlugin.h:30
void setPluginNamespace(const char *pluginNamespace) override
Set the namespace that this plugin object belongs to.
Definition: instanceNormalizationPlugin.cpp:262
def identity_engine()
Definition: trt/test_loader.py:33
std::shared_ptr< Likelihood > ptr
Definition: likelihood.h:36
void destroy() override
Destroy the plugin object.
Definition: lReluPlugin.cpp:120
bool verifyOutput(const samplesCommon::BufferManager &buffers)
Filters output detections and verify results.
Definition: sampleUffSSD.cpp:323
const char * getPluginType() const override
Return the plugin type.
Definition: split.h:110
Updates weights in an engine.
Definition: NvInferRuntime.h:828
Definition: maskRCNNKernels.h:84
int inputH
The input height.
Definition: sampleMNISTAPI.cpp:48
def infer_tacotron2_trt(encoder, decoder_iter, postnet, encoder_context, decoder_context, postnet_context, sequences, sequence_lengths, measurements, fp16)
Definition: inference_trt.py:175
const PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: regionPlugin.cpp:489
def test_raise(self)
Definition: functional_test.py:44
size_t mParamWordsize
Definition: skipLayerNormInt8InterleavedPlugin.h:80
Definition: buffers.h:207
T bitwise_cast(U u)
Definition: fp16.h:27
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: reorgPlugin.cpp:170
Type traits for floating point bits.
Definition: ieee_half.h:315
Definition: sampleMovieLensMPS.cpp:189
~BertInference()
Definition: bert_infer.h:324
input_node
Definition: test_ir.py:113
def gather(self, data, indices)
Definition: 09_shape_operations_with_the_layer_api/generate.py:41
virtual void destroy() noexcept=0
Destroy this object;.
def test_fake_quant_per_channel(self)
Definition: quant_conv_test.py:360
int mBboxesCnt
Definition: specialSlicePlugin.h:95
BatchStream(int batchSize, int maxBatches, const std::string &prefix, const std::vector< std::string > &directories)
Definition: frcnnUtils.h:222
Definition: proposalPlugin.h:125
_axis
Definition: tools/pytorch-quantization/pytorch_quantization/calib/calibrator.py:33
void cudaCheck(cudaError_t ret, std::ostream &err=std::cerr)
Definition: sampleDevice.h:28
bool clip
Definition: NvInferPluginUtils.h:96
def __init__(self, measurements, key, cpu_run=False)
Definition: Tacotron2/inference.py:165
Definition: checkMacrosPlugin.h:74
void syncAll(const TimePoint &cpuStart, const TrtCudaEvent &gpuStart, std::vector< InferenceTrace > &trace, bool skipTransfers)
Definition: sampleInference.cpp:356
int mBatchCount
Definition: BatchStream.h:375
def layer(self, inputs=[], outputs=[], *args, **kwargs)
Definition: graph.py:397
int mEosId
Definition: vocabulary.h:77
field_name
Definition: onnx-graphsurgeon/onnx_graphsurgeon/util/misc.py:72
DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes, int nbInputs) const override
Definition: detectionLayerPlugin.cpp:275
bool processInput(const samplesCommon::BufferManager &buffers)
Copies a batch of input data from SampleMovieLensParams into managed input buffers.
Definition: sampleMovieLens.cpp:333
bool verifyOutput(const samplesCommon::BufferManager &buffers) const
Verifies that the output is correct and prints it.
Definition: sampleINT8API.cpp:455
virtual IUnaryLayer * addUnary(ITensor &input, UnaryOperation operation)=0
Add a unary layer to the network.
handle
Definition: cuda.py:92
default_quant_desc_weight
Definition: _utils.py:34
virtual void reset()=0
Reset the reader position, the data reader is ready to read the data from th ebeginning again after t...
CUresult(* _cuModuleUnload)(CUmodule)
Definition: cudaDriverWrapper.h:73
virtual ~IFullyConnectedLayer()
Definition: NvInfer.h:1610
static expr negate(float arg)
Definition: samples/common/half.h:2532
Definition: tools/Polygraphy/polygraphy/backend/onnxrt/__init__.py:1
@ kSUM
Sum of the two operands.
unsigned int fused_multihead_attention_int8_384_64_kernel_sm75_cu_o_len
Definition: fused_multihead_attention_int8_384_64_kernel.sm75.cpp:2688
std::string fileName
Definition: sampleUffMaskRCNN.cpp:74
const trtcaffe::NetParameter & mMsg
Definition: caffeWeightFactory.h:58
const char * mInputBlobName
Definition: EntropyCalibrator.h:90
void print_version()
Definition: samples/common/common.h:483
void attachToContext(cudnnContext *cudnnContext, cublasContext *cublasContext, IGpuAllocator *gpuAllocator) override
Attach the plugin object to an execution context and grant the plugin the access to some context reso...
Definition: proposalPlugin.cpp:309
Definition: trt/util.py:1
void copyRNNOutputsToInputs(samplesCommon::BufferManager &buffers)
Copies Ct/Ht output from the RNN to the Ct-1/Ht-1 input buffers for next time step.
Definition: sampleCharRNN.cpp:930
virtual IRaggedSoftMaxLayer * addRaggedSoftMax(ITensor &input, ITensor &bounds)=0
Add a RaggedSoftMax layer to the network.
bool canBroadcastInputAcrossBatch(int inputIndex) const override
Definition: generateDetectionPlugin.cpp:310
Definition: batchedNMSPlugin.h:139
std::vector< float > mAnchorSizes
Definition: proposalPlugin.h:119
def exact_match_score(prediction, ground_truth)
Definition: evaluate-v1.1.py:60
void terminate() override
Release resources acquired during plugin layer initialization.
Definition: detectionLayerPlugin.cpp:136
const nvinfer1::PluginFieldCollection * getFieldNames() override
Return a list of fields that needs to be passed to createPlugin.
Definition: skipLayerNormInt8InterleavedPlugin.cpp:285
in_layers
Definition: demo/Tacotron2/waveglow/model.py:95
EnqueueImplicit(nvinfer1::IExecutionContext &context, void **buffers, int batch)
Definition: sampleInference.cpp:195
int volume
Definition: sampleUtils.h:219
virtual int32_t getTensorRTVersion() const
Return the API version with which this plugin was built.
Definition: NvInferRuntime.h:232
logger_callbacks
Definition: Polygraphy/polygraphy/logger/logger.py:135
def test_input_variable_bits(self)
Definition: quant_pooling_test.py:80
def test_no_quant(self)
Definition: quant_conv_test.py:463
std::shared_ptr< nvinfer1::ICudaEngine > mEngine
The TensorRT engine used to run the network.
Definition: sampleAlgorithmSelector.cpp:357
def test_raise(self)
Definition: quant_pooling_test.py:199
const char * getPluginVersion() const override
Return the plugin version.
Definition: generateDetectionPlugin.cpp:56
SLPAttention(ComponentWeights::ptr weights)
Definition: slpAttention.cpp:24
Definition: sampleEngines.cpp:43
virtual const char * getName() const =0
Return name of the algorithm node.
Definition: maskRCNNKernels.h:90
virtual bool isShapeTensor() const =0
Whether the tensor is a shape tensor.
def test_layer_with_strings(self)
Definition: test_ir.py:518
def get_input_metadata(self)
Definition: backend/pyt/runner.py:61
Definition: test_ir.py:110
def alignData(reference, profile, threshold)
Definition: profiler.py:136
half()
Default constructor.
Definition: ieee_half.h:1102
constexpr int32_t EnumMax< WeightsRole >()
Maximum number of elements in WeightsRole enum.
Definition: NvInferRuntime.h:696
static const float RPN_ANCHOR_SCALE
Definition: tlt_mrcnn_config.h:64
Definition: tf/test_loader.py:60
nvinfer1::Dims mInputDims
The dimensions of the input to the network.
Definition: sampleMNIST.cpp:98
Definition: nvFasterRCNNPlugin.h:103
def test_o(self)
Definition: test_ir.py:94
std::string mPrefix
Definition: pluginLogging.h:99
def malloc(self, nbytes)
Definition: cuda.py:59
def _check_dtype_matches(self, host_buffer)
Definition: cuda.py:154
static const char * CROP_AND_RESIZE_PLUGIN_NAME
Definition: cropAndResizePlugin.cpp:32
@ kDEFAULT
Register layer names in NVTX message field.
def deactivate_impl(self)
Definition: backend/trt/runner.py:176
def test_fake_quant_per_channel_other_prec(self)
Definition: quant_conv_transposed_test.py:449
def __deepcopy__(self, memo)
Definition: graph.py:447
~FCPlugin()
Definition: samples/opensource/samplePlugin/fcPlugin.h:81
std::vector< int > mInputConcatAxis
Definition: flattenConcat.h:100
1.8.17