MQL4 Random Entry Systems

[Jim]

Hi Baguette,

Hi Gurus, I wonder if you could help me with a problem. Not long ago my computer broke and I lost all the data on my hard drive including my historical data for Alpari MT4. I downloaded your EUR/USD data and was hoping someone could provide a link for GBP/USD history.

OK - Just for completeness, you can now download Alpari UK MetaTrader 4 one minute historical data for January to July 2010:

For EUR/USD here
and for GBP/USD here

Cheers,

Jim

[pkotak]

Hi Jim,

Why all those indicators? That's a good one too.

I have a "random" thought in return to play with, regarding Random EA's and Martingale in Forex. The problem is, my thought is very difficult to test without analysing over vast tracts of historical data.

It goes back to your quote about indicators. To date, I have seen no convincing arguments as to why Technical Analysis should work, although it appears to. If TA is invaild, then "all those indicators" would in fact, be useless. In addition, I conjecture that the results of both Random EA's and Martingale EA's depend on the validity of TA.
If TA is actually valid in Forex, then EA results will be random but "different" compared with the results you would get if TA is in fact retro-fitted psuedo-science. I would expect a different "profile" of results as random trades get superimposed on any TA pattern that happens to be brewing. To try and illustrate: I would expect longer sequences of wins and losses on buys and sells then on true random. Or, another example, I would expect Martingale to behave much like in a Casino if TA is invalid, but I would expect death to be either faster of slower if TA is valid. But these types of intra-patterns would only be believeable if seen in a very very large sample of results.

If on the other hand, if TA is valid (albiet for reasons unknown), then that implies it should be possible to construct/code a guaranteed winning indicator/EA.

[Jim]

Hi pkotak,

If on the other hand, if TA is valid (albiet for reasons unknown), then that implies it should be possible to construct/code a guaranteed winning indicator/EA.

Your suggestion prompted a not entirely random thought on my part. No guarantees of course, but you might like to take a look at our newly refurbished London Breakout "robot".

It uses no indicators, so that's a good start! I don't know how you define "Technical Analysis", but it does work on the assumption that certain non-random prices and times of day might be more significant than others. A few teething troubles mean it's not working awfully well just at the moment, but I have a hunch that with a little bit more work it might well prove possible to turn it into a "winning EA". Then if you wanted you could try testing it over "a very very large sample of results".

Cheers,

Jim

[matka]

Hi Jim. Very interesting idea to use randomness. Have you been considering real random number generator instead of pseudo? I think this can make a difference. Also, to achieve full randomness, I think not only entry point should be random.

I saw your post on Birt's blog so I guess you are now familiar with real tick by tick backtesting on good quality data? Have you tried it with GuruEx?

[Jim]

Hi Matka,

Hi Jim. Very interesting idea to use randomness. Have you been considering real random number generator instead of pseudo? I think this can make a difference. Also, to achieve full randomness, I think not only entry point should be random.

Thanks very much for your kind words. I did try using MetaTrader's own random number generator at one point, but it didn't seem to work terribly well. Your average digital random number generator is still strictly speaking only pseudo random though.

I do take your point about "full randomness". How far would you take the concept? Random exits? Random money management as well? I can't help thinking that such a "fully random robot" wouldn't make a lot of money if let loose on a live account though.

I saw your post on Birt's blog so I guess you are now familiar with real tick by tick backtesting on good quality data? Have you tried it with GuruEx?

I've been familiar with the concept of real tick by tick backtesting on good quality data since before MetaTrader was invented. Unfortunately for some strange reason MetaTrader still isn't very good at that sort of thing! So yes, I have tried it, but not using MT4.

Cheers,

Jim

[matka]

Jim,

Thank you for your reply.

Thanks very much for your kind words. I did try using MetaTrader's own random number generator at one point, but it didn't seem to work terribly well. Your average digital random number generator is still strictly speaking only pseudo random though.

I could not make GuruEx01A to produce different results in backtest, even with ReallyRandom set to TRUE. I have heard that MQL's MathSrand have some flaws. My friends developers helped me to amend the code and now I can use higher quality random number generator used for cryptographic purposes. I do it in the same way as Pkotak suggested by calling external dll. But as you mentioned, it is still pseudo. My idea was to use specialized hardware like computer card. However as I can not develop code myself I will probably use online solution I found on random.org which seems to be quite good, the only problem would be with extensive backtesting and per IP quota for generating numbers.

I do take your point about "full randomness". How far would you take the concept? Random exits? Random money management as well? I can't help thinking that such a "fully random robot" wouldn't make a lot of money if let loose on a live account though.

I am thinking about random exits (TP/SL) for a start. However after first experiments I realized that approach like this is totally different from "normal". For example you can not optimize parameters, entires, apply filters etc. because every single time you have different, random results. I know it sounds odd as at the end of the day we are dealing with random entry EA so what would I expect, but I only realized full extent of the issue after few experiments. This brought me the crazy idea, that the only way of improvement would be to take randomness to the limit an try to implement it properly. I do not know yet how exactly but I think money management should not be left alone.

I do agree with you that it is hard to imagine how random system would be profitable and it probably will not, but as you mentioned in your previous posts, this is scientific experiment, not a business venture ;}

I've been familiar with the concept of real tick by tick backtesting on good quality data since before MetaTrader was invented. Unfortunately for some strange reason MetaTrader still isn't very good at that sort of thing! So yes, I have tried it, but not using MT4.

I sorry Jim, I meant something different, I forgot to put "with MT4" at the end of the sentence. I was trying to say that with a bit of work it is possible to do good quality backtesting with MT4. The concept is described on Birt's webpage. Can I ask what is your preferred software for backtesting?

Cheers

Matka

[Jim]

Hi Matka

I could not make GuruEx01A to produce different results in backtest, even with ReallyRandom set to TRUE. I have heard that MQL's MathSrand have some flaws. My friends developers helped me to amend the code and now I can use higher quality random number generator used for cryptographic purposes. I do it in the same way as Pkotak suggested by calling external dll. But as you mentioned, it is still pseudo. My idea was to use specialized hardware

I had much the same problem, although sometimes ReallyRandom did seem to work as expected. I never dug deep enough to discover the cause. For the purposes of this "scientific experiment" I think specialized hardware is going a bit too far. Surely a software implementation that works will be sufficient?

I am thinking about random exits (TP/SL) for a start. However after first experiments I realized that approach like this is totally different from "normal". For example you can not optimize parameters, entires, apply filters etc. because every single time you have different, random results. I know it sounds odd as at the end of the day we are dealing with random entry EA so what would I expect, but I only realized full extent of the issue after few experiments. This brought me the crazy idea, that the only way of improvement would be to take randomness to the limit an try to implement it properly. I do not know yet how exactly but I think money management should not be left alone.

I do agree with you that it is hard to imagine how random system would be profitable and it probably will not, but as you mentioned in your previous posts, this is scientific experiment, not a business venture ;}

Please let us know what you discover!

I sorry Jim, I meant something different, I forgot to put "with MT4" at the end of the sentence. I was trying to say that with a bit of work it is possible to do good quality backtesting with MT4. The concept is described on Birt's webpage. Can I ask what is your preferred software for backtesting?

I experiment with a variety of backtesters. MT5's distributed optimization is interesting for example, though still no better than MT4 if you really need ticks. For tick based FX stuff I generally go straight to JForex in the first instance. Eventually for anything that looks particularly promising I'll use my homebrew tester, so I can be sure that identical code will be used in testing and in "real life".

Cheers,

Jim

[matka]

I had much the same problem, although sometimes ReallyRandom did seem to work as expected. I never dug deep enough to discover the cause.

The problem is with MathSrand, basically it does not work.

For the purposes of this "scientific experiment" I think specialized hardware is going a bit too far. Surely a software implementation that works will be sufficient?

Actually it is quite opposite. Even highly sophisticated _software_ solution is not even close to real random. Have a word with a skilled mathematician.

Please let us know what you discover!

I promise I will. Currently I am struggling to find time for this project. Most important, I think I found the way to optimize the performance.

Eventually for anything that looks particularly promising I'll use my homebrew tester, so I can be sure that identical code will be used in testing and in "real life".

At the moment I have a lot of doubts about this kind of approach. I am not a coder but I have a lot of experience with open source software and I strongly believe that the only way to have a properly debugged software is to make it popular and tested by as many people as possible. Sometimes very basic logical errors are almost impossible to detect by the author of the software but they can have massive impact on the credibility of the backtesting results. Can I humbly ask you for the opinion about that?

Cheers

matka

[Jim]

Hi Matka,

The problem is with MathSrand, basically it does not work.

I won't argue with that analysis!

Actually it is quite opposite. Even highly sophisticated _software_ solution is not even close to real random. Have a word with a skilled mathematician.

I understand that. I've even written original statistical research papers. However KISS is my motto. The point I'm trying to make is that for the purposes of my own experiments into "random" entries a software "pseudo random" generator (that works!) is more than sufficient. Your mileage may differ of course.

At the moment I have a lot of doubts about this kind of approach. I am not a coder but I have a lot of experience with open source software and I strongly believe that the only way to have a properly debugged software is to make it popular and tested by as many people as possible. Sometimes very basic logical errors are almost impossible to detect by the author of the software but they can have massive impact on the credibility of the backtesting results. Can I humbly ask you for the opinion about that?

I've been a professional coder for over 30 years, and I'm quite happy using code I've written myself. The ultimate test of a backtester is how closely the results it produces tie in with real trades on a live account. Any "basic logical errors" that survive that test are well worth further investigation!

Mind you I'm a fan of open source software in general too, and not only for the reason you describe. As you say, the writer of a piece of code is not really the best person to test it. With open source software if a bug is uncovered someone can fix it quickly, though that doesn't necessarily mean someone WILL fix it both adequately and quickly!

Sounds as though you use open source software to do your own backtesting. Can I humbly ask you which one?

Cheers,

Jim

[matka]

I won't argue with that analysis!

I am sorry Jim, I know I should be more specific The problem with MathStrand is that it takes current date as a rand seed so in some cases it reproduces same results.

I understand that. I've even written original statistical research papers. However KISS is my motto. The point I'm trying to make is that for the purposes of my own experiments into "random" entries a software "pseudo random" generator (that works!) is more than sufficient. Your mileage may differ of course.

I understand your point. What I am trying to say is with approach like that you follow "pseudo random" path, you adapt your system to kind of non random pattern which under particular circumstances can be reproduced or repeated.

Sounds as though you use open source software to do your own backtesting. Can I humbly ask you which one?

Thank you very much for your opinion. At the moment I am using MT4 with patches, not open source but fairly tested plus lots of other advantages due to it's popularity. I agree with you that ultimate test for backtester is comparison with live result tests, however sometimes it is not that easy due to non centralized nature of FX market and differences in quotations.

Cheers

matka