Class: AWSCDK::KinesisFirehose::CfnDeliveryStream::OrcSerDeProperty

Inherits:

Jsii::Struct

Object
Jsii::Struct
AWSCDK::KinesisFirehose::CfnDeliveryStream::OrcSerDeProperty

show all

Defined in:: kinesis_firehose/cfn_delivery_stream.rb

Overview

A serializer to use for converting data to the ORC format before storing it in Amazon S3.

For more information, see Apache ORC .

See Also:

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html

Instance Attribute Summary collapse

#block_size_bytes ⇒ Numeric^? readonly
The Hadoop Distributed File System (HDFS) block size.
#bloom_filter_columns ⇒ Array<String>^? readonly
The column names for which you want Firehose to create bloom filters.
#bloom_filter_false_positive_probability ⇒ Numeric^? readonly
The Bloom filter false positive probability (FPP).
#compression ⇒ String^? readonly
The compression code to use over data blocks.
#dictionary_key_threshold ⇒ Numeric^? readonly
Represents the fraction of the total number of non-null rows.
#enable_padding ⇒ Boolean, ... readonly
Set this to true to indicate that you want stripes to be padded to the HDFS block boundaries.
#format_version ⇒ String^? readonly
The version of the file to write.
#padding_tolerance ⇒ Numeric^? readonly
A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size.
#row_index_stride ⇒ Numeric^? readonly
The number of rows between index entries.
#stripe_size_bytes ⇒ Numeric^? readonly
The number of bytes in each stripe.

Class Method Summary collapse

.jsii_properties ⇒ Object

Instance Method Summary collapse

#initialize(block_size_bytes: nil, bloom_filter_columns: nil, bloom_filter_false_positive_probability: nil, compression: nil, dictionary_key_threshold: nil, enable_padding: nil, format_version: nil, padding_tolerance: nil, row_index_stride: nil, stripe_size_bytes: nil) ⇒ OrcSerDeProperty constructor
A new instance of OrcSerDeProperty.
#to_jsii ⇒ Object

Constructor Details

#initialize(block_size_bytes: nil, bloom_filter_columns: nil, bloom_filter_false_positive_probability: nil, compression: nil, dictionary_key_threshold: nil, enable_padding: nil, format_version: nil, padding_tolerance: nil, row_index_stride: nil, stripe_size_bytes: nil) ⇒ `OrcSerDeProperty`

Returns a new instance of OrcSerDeProperty.

Parameters:

block_size_bytes (Numeric, nil) (defaults to: nil) —
The Hadoop Distributed File System (HDFS) block size.
bloom_filter_columns (Array<String>, nil) (defaults to: nil) —
The column names for which you want Firehose to create bloom filters.
bloom_filter_false_positive_probability (Numeric, nil) (defaults to: nil) —
The Bloom filter false positive probability (FPP).
compression (String, nil) (defaults to: nil) —
The compression code to use over data blocks.
dictionary_key_threshold (Numeric, nil) (defaults to: nil) —
Represents the fraction of the total number of non-null rows.
enable_padding (Boolean, AWSCDK::IResolvable, nil) (defaults to: nil) —
Set this to true to indicate that you want stripes to be padded to the HDFS block boundaries.
format_version (String, nil) (defaults to: nil) —
The version of the file to write.
padding_tolerance (Numeric, nil) (defaults to: nil) —
A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size.
row_index_stride (Numeric, nil) (defaults to: nil) —
The number of rows between index entries.
stripe_size_bytes (Numeric, nil) (defaults to: nil) —
The number of bytes in each stripe.

# File 'kinesis_firehose/cfn_delivery_stream.rb', line 3440

def initialize(block_size_bytes: nil, bloom_filter_columns: nil, bloom_filter_false_positive_probability: nil, compression: nil, dictionary_key_threshold: nil, enable_padding: nil, format_version: nil, padding_tolerance: nil, row_index_stride: nil, stripe_size_bytes: nil)
  @block_size_bytes = block_size_bytes
  Jsii::Type.check_type(@block_size_bytes, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJudW1iZXIifQ==")), "blockSizeBytes") unless @block_size_bytes.nil?
  @bloom_filter_columns = bloom_filter_columns
  Jsii::Type.check_type(@bloom_filter_columns, JSON.parse(Base64.strict_decode64("eyJjb2xsZWN0aW9uIjp7ImVsZW1lbnR0eXBlIjp7InByaW1pdGl2ZSI6InN0cmluZyJ9LCJraW5kIjoiYXJyYXkifX0=")), "bloomFilterColumns") unless @bloom_filter_columns.nil?
  @bloom_filter_false_positive_probability = bloom_filter_false_positive_probability
  Jsii::Type.check_type(@bloom_filter_false_positive_probability, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJudW1iZXIifQ==")), "bloomFilterFalsePositiveProbability") unless @bloom_filter_false_positive_probability.nil?
  @compression = compression
  Jsii::Type.check_type(@compression, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJzdHJpbmcifQ==")), "compression") unless @compression.nil?
  @dictionary_key_threshold = dictionary_key_threshold
  Jsii::Type.check_type(@dictionary_key_threshold, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJudW1iZXIifQ==")), "dictionaryKeyThreshold") unless @dictionary_key_threshold.nil?
  @enable_padding = enable_padding
  Jsii::Type.check_type(@enable_padding, JSON.parse(Base64.strict_decode64("eyJ1bmlvbiI6eyJ0eXBlcyI6W3sicHJpbWl0aXZlIjoiYm9vbGVhbiJ9LHsiZnFuIjoiYXdzLWNkay1saWIuSVJlc29sdmFibGUifV19fQ==")), "enablePadding") unless @enable_padding.nil?
  @format_version = format_version
  Jsii::Type.check_type(@format_version, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJzdHJpbmcifQ==")), "formatVersion") unless @format_version.nil?
  @padding_tolerance = padding_tolerance
  Jsii::Type.check_type(@padding_tolerance, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJudW1iZXIifQ==")), "paddingTolerance") unless @padding_tolerance.nil?
  @row_index_stride = row_index_stride
  Jsii::Type.check_type(@row_index_stride, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJudW1iZXIifQ==")), "rowIndexStride") unless @row_index_stride.nil?
  @stripe_size_bytes = stripe_size_bytes
  Jsii::Type.check_type(@stripe_size_bytes, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJudW1iZXIifQ==")), "stripeSizeBytes") unless @stripe_size_bytes.nil?
end

Instance Attribute Details

#block_size_bytes ⇒ `Numeric`^? (readonly)

The Hadoop Distributed File System (HDFS) block size.

This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is 256 MiB and the minimum is 64 MiB. Firehose uses this value for padding calculations.

Returns:

(Numeric, nil)

See Also:

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-blocksizebytes



3469
3470
3471

# File 'kinesis_firehose/cfn_delivery_stream.rb', line 3469

def block_size_bytes
  @block_size_bytes
end

#bloom_filter_columns ⇒ `Array<String>`^? (readonly)

The column names for which you want Firehose to create bloom filters.

The default is null .

Returns:

(Array<String>, nil)

See Also:

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-bloomfiltercolumns



3476
3477
3478

# File 'kinesis_firehose/cfn_delivery_stream.rb', line 3476

def bloom_filter_columns
  @bloom_filter_columns
end

#bloom_filter_false_positive_probability ⇒ `Numeric`^? (readonly)

The Bloom filter false positive probability (FPP).

The lower the FPP, the bigger the Bloom filter. The default value is 0.05, the minimum is 0, and the maximum is 1.

Returns:

(Numeric, nil)

See Also:

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-bloomfilterfalsepositiveprobability



3483
3484
3485

# File 'kinesis_firehose/cfn_delivery_stream.rb', line 3483

def bloom_filter_false_positive_probability
  @bloom_filter_false_positive_probability
end

#compression ⇒ `String`^? (readonly)

The compression code to use over data blocks.

The default is SNAPPY .

Returns:

(String, nil)

See Also:

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-compression



3490
3491
3492

# File 'kinesis_firehose/cfn_delivery_stream.rb', line 3490

def compression
  @compression
end

#dictionary_key_threshold ⇒ `Numeric`^? (readonly)

Represents the fraction of the total number of non-null rows.

To turn off dictionary encoding, set this fraction to a number that is less than the number of distinct keys in a dictionary. To always use dictionary encoding, set this threshold to 1.

Returns:

(Numeric, nil)

See Also:

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-dictionarykeythreshold



3497
3498
3499

# File 'kinesis_firehose/cfn_delivery_stream.rb', line 3497

def dictionary_key_threshold
  @dictionary_key_threshold
end

#enable_padding ⇒ `Boolean`, ... (readonly)

Set this to true to indicate that you want stripes to be padded to the HDFS block boundaries.

This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is false .

Returns:

(Boolean, AWSCDK::IResolvable, nil)

See Also:

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-enablepadding



3504
3505
3506

# File 'kinesis_firehose/cfn_delivery_stream.rb', line 3504

def enable_padding
  @enable_padding
end

#format_version ⇒ `String`^? (readonly)

The version of the file to write.

The possible values are V0_11 and V0_12 . The default is V0_12 .

Returns:

(String, nil)

See Also:

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-formatversion



3511
3512
3513

# File 'kinesis_firehose/cfn_delivery_stream.rb', line 3511

def format_version
  @format_version
end

#padding_tolerance ⇒ `Numeric`^? (readonly)

A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size.

The default value is 0.05, which means 5 percent of stripe size.

For the default values of 64 MiB ORC stripes and 256 MiB HDFS blocks, the default block padding tolerance of 5 percent reserves a maximum of 3.2 MiB for padding within the 256 MiB block. In such a case, if the available size within the block is more than 3.2 MiB, a new, smaller stripe is inserted to fit within that space. This ensures that no stripe crosses block boundaries and causes remote reads within a node-local task.

Kinesis Data Firehose ignores this parameter when EnablePadding is false .

Returns:

(Numeric, nil)

See Also:

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-paddingtolerance



3522
3523
3524

# File 'kinesis_firehose/cfn_delivery_stream.rb', line 3522

def padding_tolerance
  @padding_tolerance
end

#row_index_stride ⇒ `Numeric`^? (readonly)

The number of rows between index entries.

The default is 10,000 and the minimum is 1,000.

Returns:

(Numeric, nil)

See Also:

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-rowindexstride



3529
3530
3531

# File 'kinesis_firehose/cfn_delivery_stream.rb', line 3529

def row_index_stride
  @row_index_stride
end

#stripe_size_bytes ⇒ `Numeric`^? (readonly)

The number of bytes in each stripe.

The default is 64 MiB and the minimum is 8 MiB.

Returns:

(Numeric, nil)

See Also:

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-stripesizebytes



3536
3537
3538

# File 'kinesis_firehose/cfn_delivery_stream.rb', line 3536

def stripe_size_bytes
  @stripe_size_bytes
end

Class Method Details

.jsii_properties ⇒ `Object`

# File 'kinesis_firehose/cfn_delivery_stream.rb', line 3538

def self.jsii_properties
  {
    :block_size_bytes => "blockSizeBytes",
    :bloom_filter_columns => "bloomFilterColumns",
    :bloom_filter_false_positive_probability => "bloomFilterFalsePositiveProbability",
    :compression => "compression",
    :dictionary_key_threshold => "dictionaryKeyThreshold",
    :enable_padding => "enablePadding",
    :format_version => "formatVersion",
    :padding_tolerance => "paddingTolerance",
    :row_index_stride => "rowIndexStride",
    :stripe_size_bytes => "stripeSizeBytes",
  }
end

Instance Method Details

#to_jsii ⇒ `Object`

# File 'kinesis_firehose/cfn_delivery_stream.rb', line 3553

def to_jsii
  result = {}
  result.merge!({
    "blockSizeBytes" => @block_size_bytes,
    "bloomFilterColumns" => @bloom_filter_columns,
    "bloomFilterFalsePositiveProbability" => @bloom_filter_false_positive_probability,
    "compression" => @compression,
    "dictionaryKeyThreshold" => @dictionary_key_threshold,
    "enablePadding" => @enable_padding,
    "formatVersion" => @format_version,
    "paddingTolerance" => @padding_tolerance,
    "rowIndexStride" => @row_index_stride,
    "stripeSizeBytes" => @stripe_size_bytes,
  })
  result.compact
end

Class: AWSCDK::KinesisFirehose::CfnDeliveryStream::OrcSerDeProperty

Overview

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(block_size_bytes: nil, bloom_filter_columns: nil, bloom_filter_false_positive_probability: nil, compression: nil, dictionary_key_threshold: nil, enable_padding: nil, format_version: nil, padding_tolerance: nil, row_index_stride: nil, stripe_size_bytes: nil) ⇒ OrcSerDeProperty

Instance Attribute Details

#block_size_bytes ⇒ Numeric? (readonly)

#bloom_filter_columns ⇒ Array<String>? (readonly)

#bloom_filter_false_positive_probability ⇒ Numeric? (readonly)

#compression ⇒ String? (readonly)

#dictionary_key_threshold ⇒ Numeric? (readonly)

#enable_padding ⇒ Boolean, ... (readonly)

#format_version ⇒ String? (readonly)

#padding_tolerance ⇒ Numeric? (readonly)

#row_index_stride ⇒ Numeric? (readonly)

#stripe_size_bytes ⇒ Numeric? (readonly)

Class Method Details

.jsii_properties ⇒ Object

Instance Method Details

#to_jsii ⇒ Object

#initialize(block_size_bytes: nil, bloom_filter_columns: nil, bloom_filter_false_positive_probability: nil, compression: nil, dictionary_key_threshold: nil, enable_padding: nil, format_version: nil, padding_tolerance: nil, row_index_stride: nil, stripe_size_bytes: nil) ⇒ `OrcSerDeProperty`

#block_size_bytes ⇒ `Numeric`^? (readonly)

#bloom_filter_columns ⇒ `Array<String>`^? (readonly)

#bloom_filter_false_positive_probability ⇒ `Numeric`^? (readonly)

#compression ⇒ `String`^? (readonly)

#dictionary_key_threshold ⇒ `Numeric`^? (readonly)

#enable_padding ⇒ `Boolean`, ... (readonly)

#format_version ⇒ `String`^? (readonly)

#padding_tolerance ⇒ `Numeric`^? (readonly)

#row_index_stride ⇒ `Numeric`^? (readonly)

#stripe_size_bytes ⇒ `Numeric`^? (readonly)

.jsii_properties ⇒ `Object`

#to_jsii ⇒ `Object`