Class: AWSCDK::Bedrock::CfnDataSource::WebCrawlerConfigurationProperty

Inherits:
Jsii::Struct
  • Object
show all
Defined in:
bedrock/cfn_data_source.rb

Overview

The configuration of web URLs that you want to crawl.

You should be authorized to crawl the URLs.

See Also:

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(crawler_limits: nil, exclusion_filters: nil, inclusion_filters: nil, scope: nil, user_agent: nil, user_agent_header: nil) ⇒ WebCrawlerConfigurationProperty

Returns a new instance of WebCrawlerConfigurationProperty.

Parameters:

  • crawler_limits (AWSCDK::IResolvable, AWSCDK::Bedrock::CfnDataSource::WebCrawlerLimitsProperty, nil) (defaults to: nil) —

    The configuration of crawl limits for the web URLs.

  • exclusion_filters (Array<String>, nil) (defaults to: nil) —

    A list of one or more exclusion regular expression patterns to exclude certain URLs.

  • inclusion_filters (Array<String>, nil) (defaults to: nil) —

    A list of one or more inclusion regular expression patterns to include certain URLs.

  • scope (String, nil) (defaults to: nil) —

    The scope of what is crawled for your URLs.

  • user_agent (String, nil) (defaults to: nil) —

    Returns the user agent suffix for your web crawler.

  • user_agent_header (String, nil) (defaults to: nil) —

    A string used for identifying the crawler or bot when it accesses a web server.



2598
2599
2600
2601
2602
2603
2604
2605
2606
2607
2608
2609
2610
2611
 
# File 'bedrock/cfn_data_source.rb', line 2598

def initialize(crawler_limits: nil, exclusion_filters: nil, inclusion_filters: nil, scope: nil, user_agent: nil, user_agent_header: nil)
  @crawler_limits = crawler_limits.is_a?(Hash) ? ::AWSCDK::Bedrock::CfnDataSource::WebCrawlerLimitsProperty.new(**crawler_limits.transform_keys(&:to_sym)) : crawler_limits
  Jsii::Type.check_type(@crawler_limits, JSON.parse(Base64.strict_decode64("eyJ1bmlvbiI6eyJ0eXBlcyI6W3siZnFuIjoiYXdzLWNkay1saWIuSVJlc29sdmFibGUifSx7ImZxbiI6ImF3cy1jZGstbGliLmF3c19iZWRyb2NrLkNmbkRhdGFTb3VyY2UuV2ViQ3Jhd2xlckxpbWl0c1Byb3BlcnR5In1dfX0=")), "crawlerLimits") unless @crawler_limits.nil?
  @exclusion_filters = exclusion_filters
  Jsii::Type.check_type(@exclusion_filters, JSON.parse(Base64.strict_decode64("eyJjb2xsZWN0aW9uIjp7ImVsZW1lbnR0eXBlIjp7InByaW1pdGl2ZSI6InN0cmluZyJ9LCJraW5kIjoiYXJyYXkifX0=")), "exclusionFilters") unless @exclusion_filters.nil?
  @inclusion_filters = inclusion_filters
  Jsii::Type.check_type(@inclusion_filters, JSON.parse(Base64.strict_decode64("eyJjb2xsZWN0aW9uIjp7ImVsZW1lbnR0eXBlIjp7InByaW1pdGl2ZSI6InN0cmluZyJ9LCJraW5kIjoiYXJyYXkifX0=")), "inclusionFilters") unless @inclusion_filters.nil?
  @scope = scope
  Jsii::Type.check_type(@scope, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJzdHJpbmcifQ==")), "scope") unless @scope.nil?
  @user_agent = user_agent
  Jsii::Type.check_type(@user_agent, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJzdHJpbmcifQ==")), "userAgent") unless @user_agent.nil?
  @user_agent_header = user_agent_header
  Jsii::Type.check_type(@user_agent_header, JSON.parse(Base64.strict_decode64("eyJwcmltaXRpdmUiOiJzdHJpbmcifQ==")), "userAgentHeader") unless @user_agent_header.nil?
end

Instance Attribute Details

#crawler_limits ⇒ AWSCDK::IResolvable, ... (readonly)

The configuration of crawl limits for the web URLs.

Returns:

See Also:



2617
2618
2619
 
# File 'bedrock/cfn_data_source.rb', line 2617

def crawler_limits
  @crawler_limits
end

#exclusion_filters ⇒ Array<String>? (readonly)

A list of one or more exclusion regular expression patterns to exclude certain URLs.

If you specify an inclusion and exclusion filter/pattern and both match a URL, the exclusion filter takes precedence and the web content of the URL isn’t crawled.

Returns:

  • (Array<String>, nil)

See Also:



2624
2625
2626
 
# File 'bedrock/cfn_data_source.rb', line 2624

def exclusion_filters
  @exclusion_filters
end

#inclusion_filters ⇒ Array<String>? (readonly)

A list of one or more inclusion regular expression patterns to include certain URLs.

If you specify an inclusion and exclusion filter/pattern and both match a URL, the exclusion filter takes precedence and the web content of the URL isn’t crawled.

Returns:

  • (Array<String>, nil)

See Also:



2631
2632
2633
 
# File 'bedrock/cfn_data_source.rb', line 2631

def inclusion_filters
  @inclusion_filters
end

#scope ⇒ String? (readonly)

The scope of what is crawled for your URLs.

You can choose to crawl only web pages that belong to the same host or primary domain. For example, only web pages that contain the seed URL "https://docs.aws.amazon.com/bedrock/latest/userguide/" and no other domains. You can choose to include sub domains in addition to the host or primary domain. For example, web pages that contain "aws.amazon.com" can also include sub domain "docs.aws.amazon.com".

Returns:

  • (String, nil)

See Also:



2638
2639
2640
 
# File 'bedrock/cfn_data_source.rb', line 2638

def scope
  @scope
end

#user_agent ⇒ String? (readonly)

Returns the user agent suffix for your web crawler.

Returns:

  • (String, nil)

See Also:



2643
2644
2645
 
# File 'bedrock/cfn_data_source.rb', line 2643

def user_agent
  @user_agent
end

#user_agent_header ⇒ String? (readonly)

A string used for identifying the crawler or bot when it accesses a web server.

The user agent header value consists of the bedrockbot , UUID, and a user agent suffix for your crawler (if one is provided). By default, it is set to bedrockbot_UUID . You can optionally append a custom suffix to bedrockbot_UUID to allowlist a specific user agent permitted to access your source URLs.

Returns:

  • (String, nil)

See Also:



2650
2651
2652
 
# File 'bedrock/cfn_data_source.rb', line 2650

def user_agent_header
  @user_agent_header
end

Class Method Details

.jsii_properties ⇒ Object 



2652
2653
2654
2655
2656
2657
2658
2659
2660
2661
 
# File 'bedrock/cfn_data_source.rb', line 2652

def self.jsii_properties
  {
    :crawler_limits => "crawlerLimits",
    :exclusion_filters => "exclusionFilters",
    :inclusion_filters => "inclusionFilters",
    :scope => "scope",
    :user_agent => "userAgent",
    :user_agent_header => "userAgentHeader",
  }
end

Instance Method Details

#to_jsii ⇒ Object 



2663
2664
2665
2666
2667
2668
2669
2670
2671
2672
2673
2674
 
# File 'bedrock/cfn_data_source.rb', line 2663

def to_jsii
  result = {}
  result.merge!({
    "crawlerLimits" => @crawler_limits,
    "exclusionFilters" => @exclusion_filters,
    "inclusionFilters" => @inclusion_filters,
    "scope" => @scope,
    "userAgent" => @user_agent,
    "userAgentHeader" => @user_agent_header,
  })
  result.compact
end