View Full Version : Managing forex data

08-31-2011, 11:10 PM
Here's my starter for 10.
To do any form of data mining on forex data, we need need:

1) Accurate historical data, ideally tick. There is of course Metatrader data from various companies, including Alpari. A source I have used in the past is Gain Capital (http://ratedata.gaincapital.com)

2) A store for the data. We could of course just dump the data into Excel, and get surprisingly far. More realistically, to do any serious stuff, we need a SQL database. This is where things get interesting. We either hook up to the database through a programming language, OR we have to have powerful built-in procedural extensions, in which case it is perfectly feasible to manage mining data that way.

I think for us amateurs, who don't want to fork out megabucks, options are:

- MS Access: complete (if picky) SQL and good programming support (VBA), however, in my opinion Access is just too non-enterprise.

- MySQL (open source): free, fast and no size limit. However, SQL limited outside of standard operations and no Analytics which is a real pain. Stored proc language is far too limited to do anything serious. Good choice if willing to hook up and program via your fav programming language.

- SQL Server Express: comprehensive SQL + T-SQL procedural, powerful, but size limited.

- Oracle Xpress: comprehensive SQL (with extensive Analytics) + PL/SQL procedural + extensive and very powerful built-in packages, but sadly, size limited.

- Postgress (open source): I have never used this, but have heard that the DB is almost as stable as MySQL, and has a PL/SQL like programming language. Probably worth investigating.

For what its worth, I use Oracle Xpress (because I love the Analytics + the built-in packages) or I go to MySQL if I hit size limits.

What's the DB of choice for others doing forex data analysis?

09-01-2011, 08:12 AM
Hi Prashant,

Personally I use MySQL, mostly on a server somewhere in the clouds. I have an irrational fear of using software for which I do not possess the source code. Especially if it says Microsoft on the label!

Mind you, I don't it use for any complex data analysis. Not yet at least!



09-05-2011, 09:29 PM

let me suggest freeware: openoffice/libreoffice calc or, maybe better suited, gnumeric (http://projects.gnome.org/gnumeric/) .

If you want a database, I'd also venture for MySQL, as it's easily available/installed as part of a "LAMP" or "WAMP (http://www.wampserver.com/en/download.php)" package.

I think that generally freeware is better suited as it's simpler to get more people involved in contributing/backtesting.


09-05-2011, 10:22 PM
Hello pippa

Both SQL Server Express and Oracle XE are also free, but size limited.
There are ways round this (http://techxplorer.com/2010/10/05/exploring-oracle-database-xe-in-an-ubuntu-virtual-machine/) (with Oracle at least), with multiple installs on a Linux VM, creating a powerful nest of distributed DBs on a single physical machine, but somewhat complex to setup, and in fact, program/use. The enterprise versions of both DBs are expensive, of course.
MySQL's great, and I use it with perl or .NET, however there's no denying that the SQL on offer is primitive in comparision.

09-06-2011, 12:26 AM
Well, thanks to your answer, I checked out the link. Interesting.

For people with better understanding/higher capabilities than my 'copy & paste' trials,

Stanford University offers free courses (via web/video) on
Artifical Intelligence (http://www.ai-class.com/)and
Introduction to Databases (http://db-class.org/).

09-07-2011, 11:13 AM
Hi Pippa,

Stanford University offers free courses (via web/video) on Artificial Intelligence and Introduction to Databases.

Thanks very much for that most interesting piece of news. It seems there's also a course on Machine Learning, which Ray is very interested in (http://trading-gurus.com/artificial-intelligence-or-artificial-stupidity/) learning more about himself.



07-22-2012, 05:35 AM
Feel comfortable while work with MySQL to manage Forex data !! i am not trying those software which i do not posses the source code and installing problems. Business Management Course Online (http://www.coursesonline.com.au/)

08-01-2012, 04:40 PM
Just a quick point - Postgres is much more than almost as stable as MySQL.

If I needed to use open source I would choose Postgres over mysql everytime.. I would consider it over MsSQL in certain instances too.