Free version available
Magnifying Glass

Blog archive for October, 2008

Syndicated Posts

FaceBook

Twitter

Standards for exchanging financial data

Thursday, October 2nd, 2008 by Jason

So today I completed our OFX (Open Financial Exchange) file import option, a task made monumentally simpler thanks to Andy Smith’s handy ruby library (thanks Andy!). I’ll explain what this is all about.

PocketSmith will take an import of your financial transactions from your bank or credit card provider, and crunch/sort/filter/make it dance for you. We’ve had fun reinterpreting the data so far, however we’ve really just gotten started as we have further plans to implement a range of innovative ways of getting the information to you. Simply put, we’re building in features that we want our respective online banking interfaces to have; the primary requirement being the ability to get the information we want as quickly and as easily as possible.

One of the challenges for us in making this feature accessible to the global community is the lack of common standards for exchanging financial data. The ’staple’ export from most online banking facilities is the CSV (comma-separated value) file. It’s a simple format – just a text file with comma-delimited values –  that can be opened in a text editor, or an application such as Microsoft Excel in which it resembles a spreadsheet, however no two banks in New Zealand generate CSV files the same way. Some have header rows describing the account, and others don’t; the number of columns differ; the content within the columns differ; the dates are formatted differently; the list goes on.

As you can see, it’s not an ideal – nor productive – method for exchanging data. Certainly not for developers, anyway. Worst of all, the integrity of the data is not robust. For example, open and re-save a CSV in Excel and risk altering the contents of the file, making it unrecognisable by the importing application (Excel has a bad habit of changing the date formats in CSVs).

The first import feature built into PocketSmith takes CSVs from all New Zealand banks – so the application has to first analyse which bank a CSV has come from, after which it refactors and imports the data into the user’s PocketSmith account. The reason we take the trouble to do this is to make the feature as simple to use as possible for the user: insert file here, don’t worry about telling us how your CSV is formatted, or which bank it came from.

This has a number of clear limitations of course: we’d have to write code to suit every single bank we want to get data from. Surely there must be a better way!

Well there is. Financial data comes in other formats, two of the more popular ones are OFX (as described above) and QIF (Quicken Interchange Format). OFX is XML-based and therefore well-suited to streamlining via web services or some RESTful interface – which is the next big step for PocketSmith (hello banks!).

Even so, I suspect that the financial data saga won’t go away quite yet. As the feature was rolled into the beta tonight I glumly discovered that one local bank’s OFX file format isn’t quite kosher and therefore won’t be imported; and another offers CSVs and QIFs for download – but not OFX. Also, we’re getting the odd wibble when someone uploads a corrupt CSV, which must be confusing for a user who probably isn’t aware that they might’ve somehow tampered with the fragile nature of the file.

Someday (hopefully soon) we’ll get to easily streamline NZ transactions – read-only, of course – directly into PocketSmith from users’ bank accounts. Which would surely make life much more pleasant for all and sundry, in particular, yours truly.

Getting SSL working with rails, nginx, thin and Slicehost

Wednesday, October 1st, 2008 by James

As PocketSmith deals with confidential calculations and data, having the entire application run under a secure, encrypted connection has always been the intention. In a general sense, this means that the data transferred between the user’s machine and the application server can only be read by the user or the server. This going last night, and the below are my notes from getting everything running under a secure connection.

We have a virtual server with Slicehost, running the latest Ubuntu server (8.04) with the application being served with a Thin cluster on Nginx. The below is the process I went through to get everything going, both on our test virtual machine and beta production environment.

Server Setup

I was considering actually writing a post on this alone, but the relative ease of the setup does not warrant a separate post. The setup of the server used wholly the tutorials provided by Slicehost – they are awesome. I was completely fresh to Linux, however they were extremely straight-forward with specific commands in an easy to read format. Here they are in the order I followed them in when I set up the server about a month ago – follow this and any newbie will be able to nail it.

Getting SSL Going

1) Install OpenSLL

sudo aptitude install openssl

2) Choose a certificate provider

If you are just looking at getting SSL set up locally, you do not need to buy a certificate, there is a gem for the creation of a self-signed certificate. In this case I was setting up an actual certificate, however you can find instructions from Slicehost right here.

We decided to go with a GeoTrust certificate, because the prices were pretty sharp and the button is not as ugly as other alternatives. How shallow.

3) Follow these instructions here

These were the most descriptive but concise instructions that I found for generating the information required to get a certificate. I followed this up to the copying of the key / certificate.

http://docs.railsboxcar.com/Nginx_with_SSL

4) Get rails ready for SSL

Firstly, install the SSL requirement plugin. Go to your rails application directory, then give it a:

script/plugin install ssl_requirement

Then add the line

include SslRequirement

Into your app/controllers/application.rb file.

5) Decide what you want secure

You can add a simple line at the top of each individual controller to determine what actions you want to be secure, as described here: http://squeejee.com/articles/12-rails-nginx-and-ssl. However we wanted to make all pages in the application secure, so we added the following to app/controllers/application.rb.

def ssl_required?
  if ENV["RAILS_ENV"] == "production"
    true
  end
end

This then makes all pages require a secure connection, however only when the servers are started in the production environment.

6) Configure Nginx – /etc/nginx/sites-available/the.websiteurl.com

This is where things got a bit tricky for us, with the identical virtual machine and the live server behaving differently. What we needed to do however was to accept all connections over https, and redirect all attempted connections to http to the secure version. We already had a Nginx configuration file that we set up to respond to only one domain name (i.e. not both pocketsmith.com and www.pocketsmith.com), based on the Slicehost tutorial linked above.

Then through blending other tutorials we ended up with the following Nginx configuration (in /etc/nginx/sites-available/the.websiteurl.com). If you use the below, just replace anything bold.

upstream domain1 {
        server 127.0.0.1:3000;
        server 127.0.0.1:3001;
    }

server {
    listen               80;
    rewrite ^(.*)        https://the.websiteurl.com$1 redirect;
}

server {
            listen   443;
	    ssl on;

	    ssl_certificate /usr/local/nginx/certs/cert-name-goes-here.crt;
	    ssl_certificate_key /usr/local/nginx/certs/key-name-goes-here.key;

            server_name the.websiteurl.com;

            access_log /home/username/path_to_application_folder/log/access.log;
            error_log /home/username/path_to_application_folder/log/error.log;

            root   /home/username/path_to_application_folder/public/;
            index  index.html;

            location / {
			  proxy_set_header X_FORWARDED_PROTO https;

                          proxy_set_header  X-Real-IP  $remote_addr;
                          proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
                          proxy_set_header Host $http_host;
                          proxy_redirect false;

                          if (-f $request_filename/index.html) {
                                           rewrite (.*) $1/index.html break;
                          }

                          if (-f $request_filename.html) {
                                           rewrite (.*) $1.html break;
                          }

                          if (!-f $request_filename) {
                                           proxy_pass http://domain1;
                                           break;
                          }

            }
}

Note that we are listening on port 80 right at the top of the file, and performing a redirect to the secure version of the page. This took way to long for me to work out.

7) Panic and flee to your business partners house

So just as you are about to save this critical file and making a time-critical change on your live server, you can expect that your internet connection will drop and won’t come back. This means you quickly leave, get to an internet connection and hope that everything hasn’t melted. Nothing like compounding the intensity of the situation heh.

8 ) Battle with inexplicable differences between staging and production servers

The configuration that worked above on the staging server simply broke when put on the identical production server. The user was not being redirected from the insecure connection to the secure one, with the port 80 configuration – it appeared that Nginx was using the default configuration instead of the.websiteurl.com. The quick and dirty fix was to comment everything in the default configuration out, and put the port 80 server into the default. Although this does mean that we are up, secure and able to move onto other pressing things, I need to look further into what the cause for this discrepancy between staging and live is.

Despite the hiccups, we are running everything over a secure connection with no issues. Success!

Let’s get organised :)

Wednesday, October 1st, 2008 by Francois

The more tasks we have, the more it gets complicated to keep track of our actions. Let me explain that to you with a simple example:

  • James is sending a mail to the editor of a local newspaper. At the same time, he gets a phone call from one of his contacts and fixes a feature while talking. After dealing with these issues, James wants to finish his email but gets distracted by…
  • …Francois who wants to get James’ IRD number. Francois is also putting together a mailing which will keep people connected to the PocketSmith community. Then, Francois is afraid that…
  • …Jason will come back to him with the question: “Hey you Frenchman, how many people did we send an invitation to?”. Before I can even say something (in French btw :p), Jason would already have moved on and gotten the PDF feature fixed (and wrote down one or two mails to the Malaysian Mafia ;)

To sum up, we are all pretty flat out at the moment, especially due to the diverse nature of the nature of our tasks (dev, design, marketing, sales, pr, business development…) SO, I figured out that because I am (definitely) not a tech guy (even I DID get the story about migrating the app to the server running Ubuntu!), I would better ensure that these guys can switch from one task to another as quickly as possible. The good point is, a lot of tools are there to help me doing that. Being Google Apps users, we have access to a large range of free online tools. I am not a big fan of monopolies but I must admit Google Apps are simple and useful.

However, something I have learned is whatever tools you have, it does not do the magic by itself. As a  consequence, I do have to update the CRM and feedback documents regularly to make sure we got the last piece of information available. Having worked a bit around KM (Knowledge Management) theories, I will been keen on putting together a Google Site to create a place to share useful information and enhance collaboration (wootwoot :)  That’s gonna be for tomorrow then!

If you would like to share your experience with structuring your startup or small business, please leave a comment!