Downloading 500,000 Amazon URLS per hour

[futbolplayer]

How are these large retail arbitrage companies managing to download 500,000 urls per hour?

DSM tool for example,

reprices every hour. Has 10,000 users, and let's conservatively say all these users have combined 500,000 urls. Any ideas into how they are doing this and keeping it cost efficient?

Here is an example.

A linode box that costs $10.00 month can manage 2 selenium processed concurrently. Each selenium process can download at a rate of about 1.333 URL/minute. So,

For $10.00 a month, you can download ~160 URLS per hour.

Obviously as system architecture increases download speed should too as well as numbers of CPUs, however it would still be incredibly expensive to download this many URLS.

Thoughts?

[Canadianleaf]

im confused

[futbolplayer]

Okay, large retail arbitrage companies web scrape to get supplier data (i.e - prices, quantities, etc..). They have a large number of URLS they download in a very short amount of time. How are they doing this in a cost effective manner? More specifically, how are they downloading so many pages so fast. Even with many concurrent processed to download the information, it would be very expensive.

[Canadianleaf]

Im not sure, but this is something Id be interested in doing myself. I have lists of ASIN's and items I track manually (and with the help of Keepa).

If I could find a way to track ASIN's and show the sellers currently selling and their quantities that would be useful. Keepa currently only lists the buybox holder sellers name.

If I could get a service like this id pay for it.

[saiko1]

Originally Posted by futbolplayer

can download at a rate of about 1.333 URL/minute.

Uhh, why on Earth would this be so slow? Ignoring the fact that there are APIs for a lot of this, if you're scraping, even via unreliable proxies in far out countries, there's no way this should take that long to scrape one URL.

[Fas]

Am sure there can be bots built especially for this purpose.

I once myself designed a small JAVA bot to extract certain category products, and I was able to get 100s of urls in few minutes.

Ofcourse my bot crashes at 150+ of so, but I was ok with the results.

So, harvesting urls through an extractor bot specifically built for this purpose maybe what you are looking at.

[futbolplayer]

Because each IP address can only hit Amazon at an average rate of 1 url per 45 seconds or it will become blocked pretty quickly.

[futbolplayer]

Amazons API does not give you quantities.

[saiko1]

Originally Posted by futbolplayer

Because each IP address can only hit Amazon at an average rate of 1 url per 45 seconds or it will become blocked pretty quickly.

Ah, well there's the problem. If you want to scrape Amazon, you need to do it from multiple IPs. Last time I threw together something for this I just had it pull lists of random open proxies and use them. Rotate through them and run requests in parallel.

[agent006140]

I heard there are software which not just track,but list them on Ebay for you and constantly adjust your quantity and asking price,real time.
But they are expensive and Ebay may have already banned it.
Go to redit or reddit and ask,
https://www.reddit.com/r/EbayDropshippingGuide/