What Shopping Websites Would You Like Cataloged/Scraped?

Unrvl on 03/07/2015 - 21:20

Hi all,

I can never find a bargain, always too late to it and just generally suck at finding bargains however i would like to give back to this community since it's solely responsible for making my wallet upset.

What i do have is a particular set of skills yes i did that - programming

as others have done in the past, i'm going to build a scraper for certain websites and grab all the item data daily and compare them to the previous days prices, then display pretty percentage drops etc. finally i will be able to contribute here!

I am pretty sure there's a lot of people who have a scraper going for woolies/coles/dicksmith any of the big guns. so i'd like you to give me suggestions on any other websites. most recommended will get higher priorities.

thanks!

General Discussion

Comments

  • scrimshaw on 03/07/2015 - 21:41

    Price searching exists (Static Ice is pretty good) but a price 'tracker' doesn't yet. We had PriceSlice and Mild Surprise, but they went offline as of 2013-2014 and we no longer hear from the developers.

    Ozbargain currently has integration with CamelCamelCamel but this obviously only tracks Amazon prices. But it'd be nice if a tracker exists for all the big-box electronics / appliance stores like JB, Dick, OW and Harvey.

    • Unrvl on 03/07/2015 - 21:48

      hmm i recall a post awhile back where someone cataloged dick smith and showed prices over a week, with new percentage drops etc. ideally this is what id like to do, but with lots of different sites that show huge price drops etc so i can be more useful.

      i have chosen shoppingexpress as my first target and trial for my framework in place, once im happy with that i will create plugins for lots of different websites to monitor, hence why im asking for ideas. i thought people had the big box stores already down.

      im thinking if i make a nice framework and allow other developers to create plugins with a small amount of code to parse each website, we could all band together and make an awesome tracker.

  • Unrvl on 03/07/2015 - 23:23

    I have parsed all of shopping express's items and logged them to my database, only took 10 minutes of code and 5 minutes to fetch all the data. once i have my framework in place that scans the database and monitors change in price, and the front end website code to display findings, it should be really easy to start a nice collection.

  • Ughhh on 03/07/2015 - 23:23

    All these price comparison sites seem to cater for electronics. Something new would be grocery items and skin care/beauty/cosmetic items (high end and drugstore).

    • Unrvl on 03/07/2015 - 23:25

      websites like shopbot, static ice etc simply show current price of items compared to other websites. im planning to have all items from a bunch of different online shops and a pretty website to show each days biggest price drops in different views.

      • scotty on 04/07/2015 - 17:00

        I talked to Neil from Shopbot last year asking them whether they have such feature since they are already getting data feed from merchants (much better than scraping). I asked because ask the other price tracking sites have folded. They told me they already have the capacity, but I guess it is probably not a good enough business model to them.

        • Unrvl on 04/07/2015 - 17:03

          maybe you could extend OzBargain to include this type of software that monitors changes in prices to automatically pump out some bargains!

        • scotty on 04/07/2015 - 18:20

          @xEnt: yeah we did have a plan for that — all the past product deals are potentially trackable. However scraping is a grey area, and I have seen more and more merchants employing bot-checking firewall/proxy to stop scraping.

        • Unrvl on 04/07/2015 - 19:04

          @scotty: yeah but there's always ways to get around whenever they try to break the bot. even if it means indexing the the whole site over a 24 hour period and not in 5 minutes, using proxies etc. it would be pretty hard to stop a persistent bot without interrupting the public. I also think its only the big players that would go through the troubles of doing this.

          For those curious, my bot is written in java using jsoup as a html parser. i scraped shoppingexpress in about 7 lines of code, about 5 more to send to my sql db. I will have a separate program that will monitor the data in the database and store the days findings, then ill make a basic website to display said data.

        • scotty on 04/07/2015 - 19:34

          @xEnt: some sites requires JavaScript execution to solve some problems before white listing the browser session. Request pattern along is not enough these days.

          However it's best to check the sites' T&C.

  • Baysew on 04/07/2015 - 10:09
    +2

    Was hopping hoping to see which shopping websites people wanted scrapped.

    But Scraped is different to Scrapped.

  • Unrvl on 04/07/2015 - 23:59

    i have started work on the front end and monitor app. heres an early example: http://codepen.io/anon/pen/jPZgjR
    that is the overnight changes in price for Shoppingexpress.

    I also whipped up a dealsdirect scraper which, is currently scraping and takes 35 minutes with its huge 29k items. i have also found an extra value in this project, if i plan on logging up to millions of items, i'm going to need to up my game and learn new techniques/optimization which is always nice. never really had to play around with this much data.

  • Unrvl on 05/07/2015 - 13:57

    I put my first results up: https://www.ozbargain.com.au/node/201592

  • Unrvl on 06/07/2015 - 21:22

    I have made scrapers for surfstitch, oo and shoppingexpress. tracking about 100k items now! if i see anything awesome ill be sure to post it

  • Unrvl on 07/07/2015 - 12:25

    Does anyone know of any other big shops that sell all kinds of items online (like oo, dealsdirect etc)? i will add them to my list. would love to get those types of sites down before i start on specific types (clothes, food etc)

    next on my list is Kogan, then officeworks/dicksmith if nobody gives me any shop names!

Login or Join to leave a comment
Login Join