com.ibm.watson.discovery.v1.model

Class SourceOptionsWebCrawl

java.lang.Object

com.ibm.cloud.sdk.core.service.model.GenericModel

com.ibm.watson.discovery.v1.model.SourceOptionsWebCrawl

All Implemented Interfaces:

com.ibm.cloud.sdk.core.service.model.ObjectModel

public class SourceOptionsWebCrawl
extends com.ibm.cloud.sdk.core.service.model.GenericModel

Object defining which URL to crawl and how to crawl it.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	SourceOptionsWebCrawl.CrawlSpeed The number of concurrent URLs to fetch.

Constructor Summary

Constructors

Constructor and Description
SourceOptionsWebCrawl()

Method Summary

Modifier and Type	Method and Description
java.util.List<java.lang.String>	getBlacklist() Gets the blacklist.
java.lang.String	getCrawlSpeed() Gets the crawlSpeed.
java.lang.Long	getMaximumHops() Gets the maximumHops.
java.lang.Long	getRequestTimeout() Gets the requestTimeout.
java.lang.String	getUrl() Gets the url.
java.lang.Boolean	isAllowUntrustedCertificate() Gets the allowUntrustedCertificate.
java.lang.Boolean	isLimitToStartingHosts() Gets the limitToStartingHosts.
java.lang.Boolean	isOverrideRobotsTxt() Gets the overrideRobotsTxt.
void	setAllowUntrustedCertificate(java.lang.Boolean allowUntrustedCertificate) Sets the allowUntrustedCertificate.
void	setBlacklist(java.util.List<java.lang.String> blacklist) Sets the blacklist.
void	setCrawlSpeed(java.lang.String crawlSpeed) Sets the crawlSpeed.
void	setLimitToStartingHosts(java.lang.Boolean limitToStartingHosts) Sets the limitToStartingHosts.
void	setMaximumHops(long maximumHops) Sets the maximumHops.
void	setOverrideRobotsTxt(java.lang.Boolean overrideRobotsTxt) Sets the overrideRobotsTxt.
void	setRequestTimeout(long requestTimeout) Sets the requestTimeout.
void	setUrl(java.lang.String url) Sets the url.

Methods inherited from class com.ibm.cloud.sdk.core.service.model.GenericModel

equals, hashCode, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

SourceOptionsWebCrawl

public SourceOptionsWebCrawl()

Method Detail

getUrl

public java.lang.String getUrl()

Gets the url. The starting URL to crawl.

Returns:

the url

isLimitToStartingHosts

public java.lang.Boolean isLimitToStartingHosts()

Gets the limitToStartingHosts. When `true`, crawls of the specified URL are limited to the host part of the **url** field.

Returns:

the limitToStartingHosts

getCrawlSpeed

public java.lang.String getCrawlSpeed()

Gets the crawlSpeed. The number of concurrent URLs to fetch. `gentle` means one URL is fetched at a time with a delay between each call. `normal` means as many as two URLs are fectched concurrently with a short delay between fetch calls. `aggressive` means that up to ten URLs are fetched concurrently with a short delay between fetch calls.

Returns:

the crawlSpeed

isAllowUntrustedCertificate

public java.lang.Boolean isAllowUntrustedCertificate()

Gets the allowUntrustedCertificate. When `true`, allows the crawl to interact with HTTPS sites with SSL certificates with untrusted signers.

Returns:

the allowUntrustedCertificate

getMaximumHops

public java.lang.Long getMaximumHops()

Gets the maximumHops. The maximum number of hops to make from the initial URL. When a page is crawled each link on that page will also be crawled if it is within the **maximum_hops** from the initial URL. The first page crawled is 0 hops, each link crawled from the first page is 1 hop, each link crawled from those pages is 2 hops, and so on.

Returns:

the maximumHops

getRequestTimeout

public java.lang.Long getRequestTimeout()

Gets the requestTimeout. The maximum milliseconds to wait for a response from the web server.

Returns:

the requestTimeout

isOverrideRobotsTxt

public java.lang.Boolean isOverrideRobotsTxt()

Gets the overrideRobotsTxt. When `true`, the crawler will ignore any `robots.txt` encountered by the crawler. This should only ever be done when crawling a web site the user owns. This must be be set to `true` when a **gateway_id** is specied in the **credentials**.

Returns:

the overrideRobotsTxt

getBlacklist

public java.util.List<java.lang.String> getBlacklist()

Gets the blacklist. Array of URL's to be excluded while crawling. The crawler will not follow links which contains this string. For example, listing `https://ibm.com/watson` also excludes `https://ibm.com/watson/discovery`.

Returns:

the blacklist

setUrl

public void setUrl(java.lang.String url)

Sets the url.

Parameters:

url - the new url

setLimitToStartingHosts

public void setLimitToStartingHosts(java.lang.Boolean limitToStartingHosts)

Sets the limitToStartingHosts.

Parameters:

limitToStartingHosts - the new limitToStartingHosts

setCrawlSpeed

public void setCrawlSpeed(java.lang.String crawlSpeed)

Sets the crawlSpeed.

Parameters:

crawlSpeed - the new crawlSpeed

setAllowUntrustedCertificate

public void setAllowUntrustedCertificate(java.lang.Boolean allowUntrustedCertificate)

Sets the allowUntrustedCertificate.

Parameters:

allowUntrustedCertificate - the new allowUntrustedCertificate

setMaximumHops

public void setMaximumHops(long maximumHops)

Sets the maximumHops.

Parameters:

maximumHops - the new maximumHops

setRequestTimeout

public void setRequestTimeout(long requestTimeout)

Sets the requestTimeout.

Parameters:

requestTimeout - the new requestTimeout

setOverrideRobotsTxt

public void setOverrideRobotsTxt(java.lang.Boolean overrideRobotsTxt)

Sets the overrideRobotsTxt.

Parameters:

overrideRobotsTxt - the new overrideRobotsTxt

setBlacklist

public void setBlacklist(java.util.List<java.lang.String> blacklist)

Sets the blacklist.

Parameters:

blacklist - the new blacklist