Class: AWSCDK::Bedrock::CfnDataSource::SemanticChunkingConfigurationProperty

Inherits:
Jsii::Struct
  • Object
show all
Defined in:
bedrock/cfn_data_source.rb

Overview

Settings for semantic document chunking for a data source.

Semantic chunking splits a document into into smaller documents based on groups of similar content derived from the text with natural language processing.

With semantic chunking, each sentence is compared to the next to determine how similar they are. You specify a threshold in the form of a percentile, where adjacent sentences that are less similar than that percentage of sentence pairs are divided into separate chunks. For example, if you set the threshold to 90, then the 10 percent of sentence pairs that are least similar are split. So if you have 101 sentences, 100 sentence pairs are compared, and the 10 with the least similarity are split, creating 11 chunks. These chunks are further split if they exceed the max token size.

You must also specify a buffer size, which determines whether sentences are compared in isolation, or within a moving context window that includes the previous and following sentence. For example, if you set the buffer size to 1 , the embedding for sentence 10 is derived from sentences 9, 10, and 11 combined.

See Also:

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(breakpoint_percentile_threshold:, buffer_size:, max_tokens:) ⇒ SemanticChunkingConfigurationProperty

Returns a new instance of SemanticChunkingConfigurationProperty.

Parameters:

  • breakpoint_percentile_threshold (Numeric) —

    The dissimilarity threshold for splitting chunks.

  • buffer_size (Numeric) —

    The buffer size.

  • max_tokens (Numeric) —

    The maximum number of tokens that a chunk can contain.



2094
2095
2096
2097
2098
2099
2100
2101
 
# File 'bedrock/cfn_data_source.rb', line 2094

def initialize(breakpoint_percentile_threshold:, buffer_size:, max_tokens:)
  @breakpoint_percentile_threshold = breakpoint_percentile_threshold
  Jsii::Type.check_type(@breakpoint_percentile_threshold, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJudW1iZXIifQ==")), "breakpointPercentileThreshold")
  @buffer_size = buffer_size
  Jsii::Type.check_type(@buffer_size, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJudW1iZXIifQ==")), "bufferSize")
  @max_tokens = max_tokens
  Jsii::Type.check_type(@max_tokens, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJudW1iZXIifQ==")), "maxTokens")
end

Instance Attribute Details

#breakpoint_percentile_threshold ⇒ Numeric (readonly)

The dissimilarity threshold for splitting chunks.

Returns:

  • (Numeric)

See Also:



2107
2108
2109
 
# File 'bedrock/cfn_data_source.rb', line 2107

def breakpoint_percentile_threshold
  @breakpoint_percentile_threshold
end

#buffer_size ⇒ Numeric (readonly)

The buffer size.

Returns:

  • (Numeric)

See Also:



2112
2113
2114
 
# File 'bedrock/cfn_data_source.rb', line 2112

def buffer_size
  @buffer_size
end

#max_tokens ⇒ Numeric (readonly)

The maximum number of tokens that a chunk can contain.

Returns:

  • (Numeric)

See Also:



2117
2118
2119
 
# File 'bedrock/cfn_data_source.rb', line 2117

def max_tokens
  @max_tokens
end

Class Method Details

.jsii_properties ⇒ Object 



2119
2120
2121
2122
2123
2124
2125
 
# File 'bedrock/cfn_data_source.rb', line 2119

def self.jsii_properties
  {
    :breakpoint_percentile_threshold => "breakpointPercentileThreshold",
    :buffer_size => "bufferSize",
    :max_tokens => "maxTokens",
  }
end

Instance Method Details

#to_jsii ⇒ Object 



2127
2128
2129
2130
2131
2132
2133
2134
2135
 
# File 'bedrock/cfn_data_source.rb', line 2127

def to_jsii
  result = {}
  result.merge!({
    "breakpointPercentileThreshold" => @breakpoint_percentile_threshold,
    "bufferSize" => @buffer_size,
    "maxTokens" => @max_tokens,
  })
  result.compact
end