hf-price-check: Bash price checking script for Harbor Freight Coupon Database (HFQPDB.com), with XMPP notification

#hf-price-check

#Table of Contents

#Introduction

I have been a long time user (and fan!) of the excellent Harbor Freight Coupon Database website, but I wanted to improve my (previously manual) coupon searching and monitoring process.

This script is the result.

It checks that website daily (or on specified interval) and then notifies you (via XMPP message) when coupons become available for whatever lot number(s) you are interested in.

#Dependencies

bash
wget
sendxmpp
And ideally, some small and low power GNU/Linux computer that is on all the time (so we can set it up to check daily with cron). Although I suppose you could run it manually...

#How it Works

This is a (relatively) simple bash script that first searches through your todo.txt file for a certain (configurable) keyword and one or more Harbor Freight lot numbers. If no lot numbers (or keywords) are specified in the todo.txt file, that's it and the script exits.

Note: If you don't have a todo.txt file, or don't know what that is, see here.

Otherwise, if there are matches, the hfqpdb.com website will be downloaded (with wget) and then searched for any of the lot numbers you have specified.

After that (and whether there were matches or not), you will receive an XMPP message telling you what happened and also some feedback about whether everything seemed to work or not. Here is a sample message:

hf-price-check script running on <hostname>, downloading HFQPDB.com and searching it for 39 lot number(s)...
Web page downloaded successfully.
Lot no. regex test returned 100 results (if 0, the website probably changed and broke our regex).
Searched for 39 lot numbers, but no hits.

The 'regex test' is the total number of lines (results) that our lot number regex matched on the entire downloaded coupon page of hfqpdb.com. We use this as a check of the regex itself (as oppsed to the particular lot numbers you may be looking for). Just like the message says, if this returned zero, the website probably changed and our regex broke. In that case, please create a ticket or maybe even better send an email to the mailing list to bring it to everyone's attention. If I end up with any subscribers on that list, maybe I will even publish if and when I update / fix things.

N.B.: I have noticed that, on the first of the month, the count may drop significantly, some times all the way to zero. This is just a bunch of coupons expiring at once, and is completely normal. The total will slowly climb back up again each day until the following month, and the cycle will repeat.

#Configuration

Use git to clone this repository somewhere. You could also simply copy the files, however using git clone will allow you to update easier in the future, should the script break or otherwise be necessary (i.e., with a simple git pull). The default settings (and examples) assume you have cloned this project directory to ~/git/hf-price-check.
Copy hfpc.conf.dist to ~/.config/hf-price-check/hfpc.conf.
- N.B.: remove the .dist suffix.
Modify hfpc.conf to your own settings:
- Especially test_web_page_result_file. There is a sampleN.html file included within the repo, make sure this variable points to it.
- By default, testing is set to yes. This is to make sure everything is working properly before you start hammering hfqpdb.com with requests.
- Set other variables accordingly.
Make sure you install and configure all Dependencies if you did not do so already. Test and make sure they are all working. That's out of scope for this README.
Because there are a lot of moving parts here (sendxmpp, the script itself, etc.) do us all a favor and do a few test runs and make sure everything is working correctly with the sampleN.html test web page:
- Make sure you can send / receive XMPP messages properly.
- Make sure your todo_txt_file is found, and any lot numbers you put in there are recognized by the script.
- Add some lot numbers that you find in the sample web page to your todo.txt file and make sure they hit.
- Etc...
If you have any trouble at this stage, I guess send an email to the mailing list and we can see about helping and/or improving the docs.
Only after you are satisfied with all of above, change testing (in hfpc.conf) to no. Then try (only once!) to run it live, to make sure everything works.
Finally (optionally) set up cron to run the script once daily (or as often as you want (within reason!)). As always with cron, set it once initially like 5 minutes from now and make sure it fires and everything works before you set it for the middle of the night (or early morning, or whatever).

#Special Thanks

Last but certainly not least, thanks to the guy who put up hfqpdb.com and keeps it running! Without him, we wouldn't be having any Nice Things(TM) at all in the first place!

FWIW, I did reach out to the guy before I published this and told him about my script and asked him if he would mind that I share it with other people (in addition to my own use). He was OK with it, so here we are.

Maybe even consider throwing him a few bucks so he can buy himself a beer / coffee. Or click through on some of his links (I think he is ad supported), but only if you plan on actually buying something, anyway.

Note: Apparently clicking thorough on ads but not actually buying anything negatively affects his ad (payment?) rate. I learned this when visiting the HFQPDB.com site recently, where there is now a prominent notice to this effect.

#Please do not abuse!

The guy running hfqpdb.com seems like a pretty swell guy, and he is doing us all a favor (see previous section). So please don't ruin this by being a dumb ass and hammering his site with requests. For example, I run my script only once per day. With the length of time these coupons are usually valid, that should give you plenty of notice to avail yourself of any opportunities that may come of it. In fact, you could probably get away with running this script only once every 2 or even 3 days.

#What's a todo.txt file?

todo.txt is a simple text file format that is great for keeping lists like TODOs or shopping lists.

I keep my todo.txt file synchronized across all my devices with SyncThing. On Android, I can highly recommend the Simpletask Cloudless client (available in F-Droid. Anyway, I digress...

I designed this script to integrate with todo.txt, because I already use it to keep my shopping lists. So it's very handy to just add this price monitoring feature where I am already keeping my list of things to pick up at Harbor Freight.

However if you don't use todo.txt, you can simply create a plain text file like the following, and set the variable todo_txt_file to point at it. On the beginning of each line, use whatever you have set for todo_txt_keyword (here we use the default) followed by one or more lot numbers you are interested in, separated by spaces:

+HFPriceWatch: 11111 22222
+HFPriceWatch: 33333
+HFPriceWatch: 44444

Why more than one on the same line? Some times there can be more than one lot number for the same item. Usually hfqpdb.com is pretty good about listing all of them, however some times I add more than one, just to try and make sure I don't miss something.

#Changelog

2020-12-15 - v1.0.1 - Minor fixes to README only.
2020-12-15 - v1.0.0 - Final polish of README before announcing on r/harborfreight. Tag first version.
2020-11-18 - v0.0.0 - Initial upload to sr.ht (no actual version number, yet).

#Implementation Details

In case you are interested...

I wrote this in bash because it was what I was most comfortable with at the time. If I did it today I would probably use Python and something like BeautifulSoup.

The hfqpdb.com website is pretty straightforward, however, and seems to be stable. I have been running this script personally for some months before sharing it, and in that time there have been no changes I am aware of (my script has not broken yet).

There are tons of comments throughout the script, so hopefully the actual (nuts and bolts) "implememtation details" should not need any further explanation.

#A Personal Note

FWIW, this is the first project I ever pushed up to sr.ht, in fact it's the first project that I ever really shared, publicly, anywhere.

I have a few more things I would also like to share, once I've polished them up and feel they are ready.

I am really enjoying this platform so far! It is very, very refreshing having such a nice alternative to castles of JavaScript on proprietary platforms for a change! So, thanks Drew for doing what needed to be done! :)

~trs-80/hf-price-check