Automation in pattern-searching

[pkotak]

Hi Jim

I know the focus of this site is "Automation in Trading", but I have an interest in another aspect of automation in forex: hunting for patterns in pip data. If there is interest from other members in this topic, I would like to suggest setting up a section to debate ideas, ways and means.
Specifically, I am interested in:

- determining if patterns do actually exist in pip data, or are we falling for the human tendency to see patterns even when none exist

- determining if pip data behaves like other random data (statistics), and if not how does it differ

- discussing if existing computer science pattern finding mechanisms (e.g. neural nets, expert systems) can be usefully applied to pip data

- discussing search-space reduction techniques (e.g. alpha-beta pruning)

- discussing means of hunting through data with SQL and coding to determine a million different "little answers" (e.g. in an uptrend, are there more up candlesticks, or is the size of them bigger on average, or a combination of both?)

- creating retro-fitted winning stratigies


"To those who said that they had a feeling soap wasn't found in mines, the Captain had ventured to suggest that perhaps that was because no one had looked hard enough, and this possibility had been reluctantly acknowledged." THE HITCHHIKER'S GUIDE TO THE GALAXY, Chapter 32

[Jim]

Hi pkotak,

Long time no see! How are things going?

I know the focus of this site is "Automation in Trading", but I have an interest in another aspect of automation in forex: hunting for patterns in pip data. If there is interest from other members in this topic, I would like to suggest setting up a section to debate ideas, ways and means.

Sounds like a great idea to me. Anything that encompasses "automation" as opposed to "emotion" is fine with me, especially if it doesn't involve traditional "indicators". You seem to be covering a lot of bases though. Possibly a whole new section of the forum is required to cover all of them? In brief:

determining if patterns do actually exist in pip data, or are we falling for the human tendency to see patterns even when none exist

Do you mean patterns like "Tomorrow's a Wednesday in August, so the probabilities favour going to the beach instead of sitting in front of a screen all day".

Alternatively "Are pivot points just a figment of people's imagination?" or "Fractal breakouts will bring me loads of pips!"?

determining if pip data behaves like other random data (statistics), and if not how does it differ

It doesn't! See below.

discussing if existing computer science pattern finding mechanisms (e.g. neural nets, expert systems) can be usefully applied to pip data

I know one or two people who claim to make loads of pips from data mining and neural nets. When I ask for backtest results and a chance to "forward test" them things go rather quiet however. Very interesting nonetheless.

discussing search-space reduction techniques (e.g. alpha-beta pruning)

Though I do have the appropriate qualifications, I'm not a quant. I'll have to go consult some academic papers before I can talk knowledgeably on that topic!

discussing means of hunting through data with SQL and coding to determine a million different "little answers" (e.g. in an uptrend, are there more up candlesticks, or is the size of them bigger on average, or a combination of both?)

If you're comfortable with SQL, and particularly if you're interested in searching for temporal patterns in price/time information then you may wish to take a look at the AlgoTrader open source project. I'm on the development team, and so is "a small Swiss hedge fund", amongst others.

creating retro-fitted winning stratigies

Sounds as though you might mean over-optimization!

"To those who said that they had a feeling soap wasn't found in mines, the Captain had ventured to suggest that perhaps that was because no one had looked hard enough, and this possibility had been reluctantly acknowledged." THE HITCHHIKER'S GUIDE TO THE GALAXY, Chapter 32

I'm not sure that Infinite Improbability is the answer here, and for some reason I also don't think Marvin has such a good handle on things as Ray when it comes to finding a useable "edge" in the markets. Ray agreed when another hedge fund manager recently pointed out to him:

I am happy to predict that market data will continue to have extreme kurtosis and extreme variance

At any rate you've certainly sold me on the idea. What should we call this exciting new section of the forum?

Cheers,

Jim

[pkotak]

Hi Jim

...you've certainly sold me on the idea. What should we call this exciting new section of the forum

I'm grateful my suggestion is of interest, and I look forward to contributing to the new section. As for the name, how about:
"Pattern and Data mining in Forex - Disucssion about algorithms for finding patterns in pip data"

...you may wish to take a look at the AlgoTrader open source project.

At first glance, certainly of interest, especiallly the trading rules engine. I will take a detailed look at this project

...especially if it doesn't involve traditional "indicators"

I share some of your prejudice against traditional indicators, although I've had the idea for a while to create the following type of multi-indicator based trader:
- a feed of several dozen indicators, multi-timeframe, each giving a prediction (whenever they feel like it) of up/down/flat: this would be the "indicator committee"
- each "committee member" would have a dynamically determined "weight", based on their "recent performance" in predicting
- trade decisions (enter/exit) based the "committee opinion" hitting certain thresholds
This idea has some elements of an expert system trader

[Jim]

Hi pkotak,

I was wondering about something slightly more generic. What do you reckon to something like "Artificially Intelligent Trading", or does that not quite strike the note you had in mind?

"Data Mining", "Neural Networks", "Genetic Programming", "Expert Systems" and so on could then be sub-fora.

Cheers,

Jim

[pkotak]

Sounds good to me!

Regards
Prashant

[Jim]

Hi Prashant,

Here you are - The Trading Gurus Artificially Intelligent Trading forum!

Thanks very much for the suggestion. Do you think there are enough sub-fora to be going with?

Cheers,

Jim

[pkotak]

Yes indeed, and I will kick of with a post to "Data Mining", about database choices