At the end of the [[reboot]] conference this year I ended up walking to dinner with Christian Dalager and I mentioned that my “one actionable idea” contributed at the end of the conference was to create an RSS feed of my banking transactions.
As it turned out, I was talking to the right man: not only was Christian interested in this notion, but already had done work to parse his own Danish bank’s electronic data (in his case he had to screen-scrape, a considerably more ambitious task than I hoped to require).
My “bank” is actually a credit union, and my credit union uses the same MemberDirect web banking platform that many other Canadian credit unions do.
My first thought when approaching the task was to simply grab data in the handy comma-delimited ASCII file format that’s available under “Activity Format” on the “Account Activity Request” page of the site. Unfortunately the transaction data this provides only includes “metadata” — like the actual place I made a debit card payment, for example — for the current month’s transactions; before then it’s only dates and amounts. This would be fine if I wanted to start my data with the current month (I could just made sure I grabbed the data before it lost its metadata layer at the end of the month), but I want to go back in time as far as possible.
Fortunately there’s another option: MemberDirect now makes “Electronic Statements” available for download going back, in my case, to the start of 2007. While the statements are PDF files, and therefor require some additional parsing to grab transaction data, they are metadata-rich and thus the best candidate for my project.
Rather than going through the process of downloading the 29 PDF files currently available to me, I decided to write a script that would login to my MemberDirect account and grab all available statements automatically. Fortunately this proved to be quite easy: the MemberDirect authentication model simply accepts an HTTP POST with the username and password, and sends back a session cookie that can then be sent back for all future requests.
After some experimenting and parsing of the HTML that gets returned, I’ve come up with memberdirect-getstatements.php, a PHP script that uses cURL and the PHP Simple HTML DOM Parser to login, grab the index of available years, and then download all available PDF electronic statements.
This script isn’t for the faint of heart: you’ll need a PHP-equipped host with cURL (if you have a modern Mac you’ve already got both) and it’s possible that the script might need tweaking for credit unions that aren’t mine.
More information is (or shortly will be) available on this wiki page. Comments and experiences welcome.
My next step is to write a parser for the Electronic Statements that will allow me to do some basic “how much am I spending on coffee”-type data analysis.
Update: you can grab the statement parsing code here to extract transaction data from the electronic statements into a MySQL table.
Comments
Hi PeterIf you like, I could
Hi Peter
If you like, I could tell you how much you spend on coffee :)
Mehrnoosh
Hi Peter,I have tried your
Hi Peter,
I have tried your code against my credit unions (Tignish and Consolidated in Summerside). Neither of my credit unions provide electronic statements and so it’s a no go for me — too bad!
It’s odd that Tignish and
It’s odd that Tignish and Summerside don’t have this option as I assume they use the same MemberDirect system as Charlottetown.
They must use separate
They must use separate installations and maybe a different version. If you search for ‘Electronic Statements’ at Metro, the results page shows many hits for exactly what we’re looking for. The same search at Tignish / Consolidated has nothing like that. The top hits are vision statemenet and electronic services.
I can still download my account activity as a PDF, but that PDF doesn’t include the metadata outside of the past month.
Add new comment