Class: AWSCDK::KinesisFirehose::OrcOutputFormatProps

Inherits:
Jsii::Struct
  • Object
show all
Defined in:
kinesis_firehose/orc_output_format_props.rb

Overview

Props for ORC output format for data record format conversion.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(block_size: nil, bloom_filter_columns: nil, bloom_filter_false_positive_probability: nil, compression: nil, dictionary_key_threshold: nil, enable_padding: nil, format_version: nil, padding_tolerance: nil, row_index_stride: nil, stripe_size: nil) ⇒ OrcOutputFormatProps

Returns a new instance of OrcOutputFormatProps.

Parameters:

  • block_size (AWSCDK::Size, nil) (defaults to: nil) —

    The Hadoop Distributed File System (HDFS) block size.

  • bloom_filter_columns (Array<String>, nil) (defaults to: nil) —

    The column names for which you want Firehose to create bloom filters.

  • bloom_filter_false_positive_probability (Numeric, nil) (defaults to: nil) —

    The Bloom filter false positive probability (FPP).

  • compression (AWSCDK::KinesisFirehose::OrcCompression, nil) (defaults to: nil) —

    The compression code to use over data blocks.

  • dictionary_key_threshold (Numeric, nil) (defaults to: nil) —

    Determines whether dictionary encoding should be applied to a column.

  • enable_padding (Boolean, nil) (defaults to: nil) —

    Set this to true to indicate that you want stripes to be padded to the HDFS block boundaries.

  • format_version (AWSCDK::KinesisFirehose::OrcFormatVersion, nil) (defaults to: nil) —

    The version of the ORC format to write.

  • padding_tolerance (Numeric, nil) (defaults to: nil) —

    A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size.

  • row_index_stride (Numeric, nil) (defaults to: nil) —

    The number of rows between index entries.

  • stripe_size (AWSCDK::Size, nil) (defaults to: nil) —

    The number of bytes in each stripe.



17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
 
# File 'kinesis_firehose/orc_output_format_props.rb', line 17

def initialize(block_size: nil, bloom_filter_columns: nil, bloom_filter_false_positive_probability: nil, compression: nil, dictionary_key_threshold: nil, enable_padding: nil, format_version: nil, padding_tolerance: nil, row_index_stride: nil, stripe_size: nil)
  @block_size = block_size
  Jsii::Type.check_type(@block_size, JSON.parse(Base64.strict_decode64("eyJmcW4iOiJhd3MtY2RrLWxpYi5TaXplIn0=")), "blockSize") unless @block_size.nil?
  @bloom_filter_columns = bloom_filter_columns
  Jsii::Type.check_type(@bloom_filter_columns, JSON.parse(Base64.strict_decode64("eyJjb2xsZWN0aW9uIjp7ImVsZW1lbnR0eXBlIjp7InByaW1pdGl2ZSI6InN0cmluZyJ9LCJraW5kIjoiYXJyYXkifX0=")), "bloomFilterColumns") unless @bloom_filter_columns.nil?
  @bloom_filter_false_positive_probability = bloom_filter_false_positive_probability
  Jsii::Type.check_type(@bloom_filter_false_positive_probability, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJudW1iZXIifQ==")), "bloomFilterFalsePositiveProbability") unless @bloom_filter_false_positive_probability.nil?
  @compression = compression
  Jsii::Type.check_type(@compression, JSON.parse(Base64.strict_decode64("eyJmcW4iOiJhd3MtY2RrLWxpYi5hd3Nfa2luZXNpc2ZpcmVob3NlLk9yY0NvbXByZXNzaW9uIn0=")), "compression") unless @compression.nil?
  @dictionary_key_threshold = dictionary_key_threshold
  Jsii::Type.check_type(@dictionary_key_threshold, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJudW1iZXIifQ==")), "dictionaryKeyThreshold") unless @dictionary_key_threshold.nil?
  @enable_padding = enable_padding
  Jsii::Type.check_type(@enable_padding, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJib29sZWFuIn0=")), "enablePadding") unless @enable_padding.nil?
  @format_version = format_version
  Jsii::Type.check_type(@format_version, JSON.parse(Base64.strict_decode64("eyJmcW4iOiJhd3MtY2RrLWxpYi5hd3Nfa2luZXNpc2ZpcmVob3NlLk9yY0Zvcm1hdFZlcnNpb24ifQ==")), "formatVersion") unless @format_version.nil?
  @padding_tolerance = padding_tolerance
  Jsii::Type.check_type(@padding_tolerance, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJudW1iZXIifQ==")), "paddingTolerance") unless @padding_tolerance.nil?
  @row_index_stride = row_index_stride
  Jsii::Type.check_type(@row_index_stride, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJudW1iZXIifQ==")), "rowIndexStride") unless @row_index_stride.nil?
  @stripe_size = stripe_size
  Jsii::Type.check_type(@stripe_size, JSON.parse(Base64.strict_decode64("eyJmcW4iOiJhd3MtY2RrLWxpYi5TaXplIn0=")), "stripeSize") unless @stripe_size.nil?
end

Instance Attribute Details

#block_size ⇒ AWSCDK::Size? (readonly)

Note:

Default: Size.mebibytes(256)

The Hadoop Distributed File System (HDFS) block size.

This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. Firehose uses this value for padding calculations.

Returns:



47
48
49
 
# File 'kinesis_firehose/orc_output_format_props.rb', line 47

def block_size
  @block_size
end

#bloom_filter_columns ⇒ Array<String>? (readonly)

Note:

Default: no bloom filters are created

The column names for which you want Firehose to create bloom filters.

Returns:

  • (Array<String>, nil)

See Also:



53
54
55
 
# File 'kinesis_firehose/orc_output_format_props.rb', line 53

def bloom_filter_columns
  @bloom_filter_columns
end

#bloom_filter_false_positive_probability ⇒ Numeric? (readonly)

Note:

Default: 0.05

The Bloom filter false positive probability (FPP).

The lower the FPP, the bigger the bloom filter.

Returns:

  • (Numeric, nil)

See Also:



61
62
63
 
# File 'kinesis_firehose/orc_output_format_props.rb', line 61

def bloom_filter_false_positive_probability
  @bloom_filter_false_positive_probability
end

#compression ⇒ AWSCDK::KinesisFirehose::OrcCompression? (readonly)

Note:

Default: SNAPPY

The compression code to use over data blocks.

The possible values are NONE , SNAPPY , and ZLIB. Use SNAPPY for higher decompression speed. Use GZIP if the compression ratio is more important than speed.

Returns:

See Also:



71
72
73
 
# File 'kinesis_firehose/orc_output_format_props.rb', line 71

def compression
  @compression
end

#dictionary_key_threshold ⇒ Numeric? (readonly)

Note:

Default: 0.8

Determines whether dictionary encoding should be applied to a column.

If the number of distinct keys (unique values) in a column exceeds this fraction of the total non-null rows in that column, dictionary encoding will be turned off for that specific column.

To turn off dictionary encoding, set this threshold to 0. To always use dictionary encoding, set this threshold to 1.

Returns:

  • (Numeric, nil)

See Also:



81
82
83
 
# File 'kinesis_firehose/orc_output_format_props.rb', line 81

def dictionary_key_threshold
  @dictionary_key_threshold
end

#enable_padding ⇒ Boolean? (readonly)

Note:

Default: false

Set this to true to indicate that you want stripes to be padded to the HDFS block boundaries.

This is useful if you intend to copy the data from Amazon S3 to HDFS before querying.

Returns:

  • (Boolean, nil)

See Also:



89
90
91
 
# File 'kinesis_firehose/orc_output_format_props.rb', line 89

def enable_padding
  @enable_padding
end

#format_version ⇒ AWSCDK::KinesisFirehose::OrcFormatVersion? (readonly)

Note:

Default: V0_12

The version of the ORC format to write.

The possible values are V0_11 and V0_12.

Returns:

See Also:



97
98
99
 
# File 'kinesis_firehose/orc_output_format_props.rb', line 97

def format_version
  @format_version
end

#padding_tolerance ⇒ Numeric? (readonly)

Note:

Default: 0.05 if enablePadding is true

A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size.

The default value is 0.05, which means 5 percent of stripe size.

For the default values of 64 MiB ORC stripes and 256 MiB HDFS blocks, the default block padding tolerance of 5 percent reserves a maximum of 3.2 MiB for padding within the 256 MiB block. In such a case, if the available size within the block is more than 3.2 MiB, a new, smaller stripe is inserted to fit within that space. This ensures that no stripe crosses block boundaries and causes remote reads within a node-local task.

Kinesis Data Firehose ignores this parameter when EnablePadding is false .

Returns:

  • (Numeric, nil)

See Also:



111
112
113
 
# File 'kinesis_firehose/orc_output_format_props.rb', line 111

def padding_tolerance
  @padding_tolerance
end

#row_index_stride ⇒ Numeric? (readonly)

Note:

Default: 10000

The number of rows between index entries.

Returns:

  • (Numeric, nil)

See Also:



117
118
119
 
# File 'kinesis_firehose/orc_output_format_props.rb', line 117

def row_index_stride
  @row_index_stride
end

#stripe_size ⇒ AWSCDK::Size? (readonly)

Note:

Default: Size.mebibytes(64)

The number of bytes in each stripe.

The default is 64 MiB and the minimum is 8 MiB.

Returns:

See Also:



125
126
127
 
# File 'kinesis_firehose/orc_output_format_props.rb', line 125

def stripe_size
  @stripe_size
end

Class Method Details

.jsii_properties ⇒ Object 



127
128
129
130
131
132
133
134
135
136
137
138
139
140
 
# File 'kinesis_firehose/orc_output_format_props.rb', line 127

def self.jsii_properties
  {
    :block_size => "blockSize",
    :bloom_filter_columns => "bloomFilterColumns",
    :bloom_filter_false_positive_probability => "bloomFilterFalsePositiveProbability",
    :compression => "compression",
    :dictionary_key_threshold => "dictionaryKeyThreshold",
    :enable_padding => "enablePadding",
    :format_version => "formatVersion",
    :padding_tolerance => "paddingTolerance",
    :row_index_stride => "rowIndexStride",
    :stripe_size => "stripeSize",
  }
end

Instance Method Details

#to_jsii ⇒ Object 



142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
 
# File 'kinesis_firehose/orc_output_format_props.rb', line 142

def to_jsii
  result = {}
  result.merge!({
    "blockSize" => @block_size,
    "bloomFilterColumns" => @bloom_filter_columns,
    "bloomFilterFalsePositiveProbability" => @bloom_filter_false_positive_probability,
    "compression" => @compression,
    "dictionaryKeyThreshold" => @dictionary_key_threshold,
    "enablePadding" => @enable_padding,
    "formatVersion" => @format_version,
    "paddingTolerance" => @padding_tolerance,
    "rowIndexStride" => @row_index_stride,
    "stripeSize" => @stripe_size,
  })
  result.compact
end