Woolworths PriceHipster/Price History - help finding finding Woolies price history - or help web scrapping Woolies using API

Used to be a big fan of the website Price Hipster to find the special price of various items. Now it looks like Woolies has blocked them so they no longer offer price alerts or price histories for Woolies.

Does anyone know if there is an alternative website that host price history data for Woolies (or even Aldi)?

Bit of a long shot but I tried using an importXML function in Googlesheets to scrap Woolies and Coles website price data but they seem to block my attempts. If anyone knows an API or other method to get around this or some other way to log the prices into a spreadsheet I would greatly appreciate it. Just looking to find some bargains.

Example of the code I wrote that seems to now be blocked by Woolies, =IMPORTXML("https://www.woolworths.com.au/shop/productdetails/238473/colgate-plax-alcohol-free-mouthwash-freshmint","//shared-price[@class='ng-star-inserted']")

Related Stores

Woolworths
Woolworths

Comments

  • +1

    10 PRINT $7.95
    20 GOTO 10
    30 RUN

    • +3

      Syntax error in 10

      • The program never ran and line 30 is another syntax error.

  • they probably fingerprint your client, in which case you'll need to use selenium or something similar

    • That won't help. They block IP addresses too. Just from browsing around a lot at home you will get blocked at your home. Just be patient. They'll stop being stupid eventually. I've seen this many times before with many different retailers.

      • having a dynamic ip address has some advantages :)

  • +1

    You can just use their API (238473) was from the URL known as the stockcode:
    https://www.woolworths.com.au/apis/ui/product/detail/238473

  • Yes, you can use their API. Ihave created a Google Apps Script function to pull the price and various other pieces of information from their API, feel freet o use the below.

    function WOWData(Article, Attribute){
      var url = "https://www.woolworths.com.au/apis/ui/products/" + Article;
      var result = UrlFetchApp.fetch(url).getContentText();
      var JSONResponse = JSON.parse(result);
      if (JSONResponse.pageCount===0) {
        return "";
      }
      else {
        switch (Attribute) {
          case "Barcode":
          return JSONResponse[0].barcode; 
          break;
          case "Price":
          return JSONResponse[0].Price; 
          break;
          case "InstorePrice:":
          return JSONResponse[0].InstorePrice; 
          break;
          case "Name":
          return JSONResponse[0].Name; 
          break;
          case "IsNew":
          return JSONResponse[0].IsNew; 
          break;
          case "IsOnSpecial":
          return JSONResponse[0].IsOnSpecial; 
          break;
          case "InstoreIsOnSpecial":
          return JSONResponse[0].InstoreIsOnSpecial; 
          break;
          case "IsEdrSpecial":
          return JSONResponse[0].IsEdrSpecial; 
          break;
          case "SavingsAmount":
          return JSONResponse[0].SavingsAmount; 
          break;
          case "WasPrice":
          return JSONResponse[0].WasPrice; 
          break;
          case "IsInStock":
          return JSONResponse[0].IsInStock; 
          break;
          case "Brand":
          return JSONResponse[0].Brand; 
          break;
          case "IsInFamily":
          return JSONResponse[0].IsInFamily; 
          break;
          case "tgawarnings":
          return JSONResponse[0].AdditionalAttributes.tgawarnings; 
          break;
          case "tgawarning":
          return JSONResponse[0].AdditionalAttributes.tgawarning; 
          break;
          case "vendorarticleid":
          return JSONResponse[0].AdditionalAttributes.vendorarticleid; 
          break;
          case "vendorcostprice":
          return JSONResponse[0].AdditionalAttributes.vendorcostprice; 
          break;
          case "Promotion":
          return JSONResponse[0].HeaderTag.Content; 
          break;
        }
      }
    }
    
  • Hi Jeremy, were you ever able to get this working? Maybe we can work on it together; I have basic Python skills for scraping if that helps.

    Edit: PS. This is a longshot, but does anybody call you "JB"?

    • Hi mate,

      I know this is an old thread, but would you happen to have any resources to help me learn how to cobble something together to price scrape some camera sites?

      • Hi, do you write in Python? Or alternatively you could try Octoparse with the free trial.

        • Cheers man, I am giving octoparse a crack. Seems good so far!

  • I wrote a code to check for my favourite items at woolies every day. My code used to call their apis/ui/product/detail to retrieve the json data, but they have implemented security checks recently. Upon closer examination, the json data is also embedded in the html source, e.g. https://www.woolworths.com.au/shop/productdetails/439768/kbs-prawn-gyoza

    It would contain this at the end of the html source:

    ....:0,&q;InstoreSavingsAmount&q;:0,&q;WasPrice&q;:22.7,&q;InstoreWasPrice&q;:22.7,&q;QuantityInTrolley&q;:0,.......

    So I would first strip off the &q;, then just look for ,FieldName:(.*), and extract the value using regex.

    • replace

      &q; -> "
      &s; -> '
      &l; -> <
      &g; -> >
      &a; -> &

      then use a json library to parse the resulting json string

      • Thanks! Do you know what those things are called? They're not standard html entities. I would've preferred to clean them up using a library rather than manually.

        • they are custom html entities that woolworths has decided to use

    • Hi mate,

      I know this is an old thread, but would you happen to have any resources to help me learn how to cobble something together similar to what you've done? It'd be for camera sites who I don't think protect their prices as vigorously as woollies seem to.

      Cheers.

      • basically you first need to figure out where the data is located. I used the browser's developer mode and look at the html source and also look at what other requests it makes, like on this screenshot: https://pasteboard.co/4P2IRcMLijB5.png

        Then just write code in your favourite language to make the same GET / POST http request. Usually the data is in json format.

  • Hi JeremyDealFiend,

    I've been working on a tool to access the Woolworths API using the R programming language. If you're interested, it's over here

Login or Join to leave a comment