Bankjob is a command line ruby program for scraping online banking sites and producing statements as local files in OFX (Open Fincancial Exchange) or CSV (Comma Separated Values) formats.
Bankjob was created for people like me who want to get their bank data into a 3rd party application but whose bank does not support downloads in OFX format.
It’s also useful for keeping a permanent store of bank statements on your computer for reading in Excel.
Mac OSX / Linux:
sudo gem install bankjob
gem install bankjob
- Scrapes an online banking website to produce a bank statement
- Stores bank statements locally in CSV files, which can be loaded directly in spreadsheets like Microsoft Excel
- Stores bank statements locally in OFX files, which can be imported by many programs such as Quicken, MS Money, Gnu Cash and uploaded to some web applications
- Built-in support for uploading to your Wesabe account (www.wesabe.com)
- Supports coding of simple rules in ruby for modifying transaction details. E.g. automatically change “pment inst 3245003” to “paid home loan interest”
bankjob --csv c:\bank\csv --scraper c:\bank\my_bpi_scraper.rb --scraper-args "<my_username> <my_password>" --wesabe "<wesabe_user> <wesabe_pass> <wesabe_acct>" --log c:\bank\bankjob.log --debug
I have this command in a .bat file which is launched daily by a scheduled task on my windows Media Center PC (which, since it’s always on and connected to the internet, makes a useful home server)
This one command will:
- scrape my online banking website after logging in as me and navigating to the page with recent transactions
- apply some rules, coded in the
my_bpi_scraper.rbfile that make the descriptions more readable
- produce a statement in comma-separated-value format, keeping the original raw data as well as the new descriptions,
storing that in a file with a name like
20090327-20090406.csvin my local directory
c:\bank\csv(serves as a sort of permanent ‘audit’ record)
- produce an OFX document with the same statement information
- upload the OFX statement to my wesabe account
- log debug-level details in bankjob.log
bankjob -h to see all of the options.
Wesabe (www.wesabe.com) is a sort of free web.2.0-money-manager-financial-social-networking application that I’m using to track my bank acounts.
I like it better than my bank’s online banking apps because it:
- shows me all my accounts at once
- lets me track certain kinds of spending, with automatically applied tags and nice graphs
- has an iphone-web-app (and a non-iphone one too)
- even has a nifty dashboard widget for Mac OS X.
- does much more than this but you should go to their site to find out more.
At this point I should point out that Bankjob and its developers (okay developer – there’s just me, I admit it. God it’s lonely here) have no affiliation with Wesabe whatsoever. I just happen to use Wesabe and like what it has to offer.
In fact I just happened to want to use Wesabe, but couldn’t because my bank doesn’t offer OFX downloads and is not supported by the automatic updater. Hence I developed Bankjob to allow me to use Wesabe.
I think there are a lot of other non-US would-be Wesabe users out there in the same boat: Wesabe and its brethren do not support our banks, and our banks don’t make it any easier by supporting OFX downloads.
There are similar services out there like Mint, Quicken Online, Yodlee and others (see one comparison here), or just google them.
The problem with most of these is that they offer even less support than Wesabe for banks outside of the US. Wesabe supports some non-US banks, but more importantly offers a public API which allows apps like Bankjob to upload accounts – among other things. (Actually I think Yodlee might have more support for non-US banks but still no public API – if your Bank is not officially supported you’re out of luck)
To use the Wesabe upload feature of Bankjob you need a Wesabe username and password. If you have more than one account with Wesabe you need to know the ID that Wesabe gives the account. This is easy to find out: if you have one a account it’s “1” and you don’t need to specify it. If you have more than one account, Bankjob will help you out. Use:
to see the general help information, but to test your Wesabe account, use
bankjob --wesabe-help "<wesabe-user-name> <wesabe-password>"
(you’ll need to use the quotes, with a space between the username and password)
With this command bankjob will list out all of your Wesabe accounts and tell you how to upload to them.
I have only one account at the moment, and here’s what happens when I use
--wesabe-help (the text in angle brackets is added by me to protect my account, but otherwise this is real input/output)
[me:~] bankjob --wesabe-help "<my-wesabe-user> <my-wesabe-password>" Connecting to Wesabe... You have 1 Wesabe accounts: Account Name: <The name of my bank account in wesabe> wesabe id: 1 account no: 0001 type: Checking balance: 2196.12 bank: <The name of my bank> To upload to this account use: bankjob [other bankjob args] --wesabe "<my-wesabe-user> <my-wesabe-password> 1" Since you have one account you do not need to specify the id number, use: bankjob [other bankjob args] --wesabe "<my-wesabe-user> <my-wesabe-password>"
Again, if in any doubt
bankjob --wesabe-help should set you straight.
Creating your own Scraper
Unfortunately Bankjob isn’t much good to you in its natural state unless you happen to bank with the same bank as me.
To make it useful you need a scraper for your own online bank site. How hard that will be depends on how handy you are with Ruby programming and with using Hpricot and Mechanize
You’ll get some help from:
- The Mechanize docs
- The Hpricot wiki
BpiScraperis included in the scrapers directory and is well documented as an example.
BaseScraperis the superclass of
BpiScraper. Using it as a base class for your own scraper is optional, but it saves you a little work by starting up Mechanize for you.
- The Scraper class itself, which you must subclass with your own Scraper, has more to say on creating your own scraper in its rubydoc
- Look for help on the forum
- Ask me directly rhubarb
Warning: Don’t upload or email anyone (including me) your private online banking information. If you create your own scraper, please share it, but remove any account details from it first.
If and when you do develop a scraper for your banksite, send it to me (without any passwords or private banking info of course!) and I’ll add it to the built-in scrapers.
How to submit patches
You can fetch the source from github at:
git clone git://github.com/rhubarb/bankjob.git
and submit any patches to the forum.
This code is free to use under the terms of the MIT license.
As far as rhubarb is concerned, this means you can:
- Use it in your private projects
- Use it in your public projects
- Use it in your commercial projects and make big bucks
- Mulch it up and use it in your garden as fertilizer
- Generally go crazy with it
Comments are welcome as are feature requests and patches.