No, you can sign up without inputting a credit card; however, your crawler will not start until you have an active subscription.

Serverless architectures—those which don't need to be hosted by always-on servers—are susceptible to influxes of spend when they experience influxes of traffic. However, this problem strictly applies to web hosting, and not web crawling.

With web hosting, you essentially pay for the bandwidth incurred by visitors that come to your site (ingress). That traffic is variable and out of your control (maybe someone posted a link to your site that went viral). While the exposure to your website is undoubtably a good thing, the month-to-month cost ends up being unpredictable, because it depends on an unknown factor.

With web crawling, you essentially pay for the requests made by your bots visiting other sites (egress). That traffic is not variable — it's totally within your control! You choose how many requests your crawlers make and how often they run. This is how your monthly spend is predictable, even with serverless architectures.

Enqueued requests are essentially outbound requests. In your code, every URL added to the queue with the enqueue() function counts as a single enqueued request.

Fetching robots.txt does not count as an enqueued request, but retries do count against the total. Enqueued requests are incurred reglardless of response status code.

Maximum concurrency is the number of allocated queue consumers multiplied by the maximum number of allowed simultaneous connections per worker (6).

In your crawler's config file, this is represented as crawl.maxConcurrency * 6.

Yes! Our upstream provider does not charge for egress, so we pass the savings on to you.

Please email [email protected] to initiate a refund request.