In the previous part, we started working on putting in order the automatic optimization conveyor, which allows us to obtain a new final EA taking into account the accumulated price data. However, we have not yet reached full automation, as difficult decisions still need to be made about how best to implement the final stages. They are difficult because if we make the wrong choice, we will have to redo a lot. Therefore, I really want to save my efforts and try to make the right choice. And nothing helps in making difficult decisions as much as… postponing them! Especially if we can afford it.
But we can postpone them in different ways. Instead of simply delaying the moment of choice, let’s try to switch to another task that will seem to allow us to get distracted, but in fact its solution can at least help increase the motivation to make a choice if not work out the right path.
The stumbling block in many debates about the use of parameter optimization is the question of how long the obtained parameters can be used for trading in the future period while maintaining the profitability and drawdown at the specified levels. And is it even possible to do this?
Although there is a popular point of view that one cannot trust the repeatability of testing results in the future, and it is only a matter of luck when the strategy “breaks down”. Probably almost all developers of trading strategies really want to believe this, otherwise the point of putting in a huge amount of effort into development and testing is lost.
Attempts to increase confidence that, by selecting good parameters, the strategy will be able to work successfully for some time have already been made repeatedly. There are articles that, in one way or another, consider the topic of periodic automatic selection of the best EA parameters. Validate EA by @fxsaber deserves a separate mention since it is precisely intended for conducting a very interesting experiment.
This tool allows us to take an arbitrary EA (the one being studied) and, having selected a certain period of time (for example, 3 years), launch the following process: the EA being studied will be optimized over a certain period (for example, 2 months), after which, using the best settings, trade in the strategy tester over a period of, say, two weeks. At the end of each two-week period, the EA being studied will again optimize for the previous two months and trade again for another two weeks. This will continue until the end of the selected 3 year interval is reached.
The end result will be a trading report showing how the EA under study would have traded over the course of all three years if it had really been periodically re-optimized and launched with updated parameters. It is clear that you can arbitrarily choose the mentioned time intervals at your own discretion. If any EA can show acceptable results with such re-optimization, then this will indicate its increased potential for use in real trading.
However, this tool has a significant limitation – the EA being studied must have open input parameters to perform optimization. If we take, for example, our final EAs obtained in the previous parts by combining many single instances, they do not have inputs that would allow them to influence the trading logic of opening positions. We will not take into account the parameters of money and risk management, since their optimization, although possible, is rather meaningless. After all, it is clear that if we increase the size of the opened positions, the result of the pass will show a greater profit, compared to what was previously obtained as a result of the pass with a smaller position size.
Therefore, let’s try to implement something similar, but applicable to our developed EAs.
In general, we need a script to fill the database with almost identical projects. The main difference will be only in the start and end dates of the optimization period. The composition of stages, stage works and tasks within the work may be completely identical. Therefore, for now, you can make a service EA with a small number of inputs, including the start date and duration of the optimization period. By running it in optimization mode with a search for start dates, we can fill the database with similar projects. It is not yet clear what other parameters make sense to include in the inputs; we will decide on them as development progresses.
Completely running all optimization tasks, even within a single project, can take a long time. If there is not one such project that needs to be completed, but a dozen or more, then we are talking about rather time-consuming tasks. Therefore, it makes sense to see if it is possible to somehow speed up the work of stage EAs. To detect bottlenecks that need to be fixed, we will use the profiler included with MetaEditor.
Next we need to decide how to simulate the work from several obtained initialization strings (each project, after completing its tasks, will provide one initialization string of the final EA). Most likely, we will need to create a new testing EA specifically designed for this type of work. But I will probably put this off until the next article.
Let’s first start by optimizing the code of the test EAs. After that, we will start creating a script for filling the database.
Before we dive into the implementation of the main task, let’s see if there is any way to speed up the code of the EAs involved in auto optimization. To detect possible bottlenecks, let’s take the final EA from the previous part for research. It combines 32 instances of single trading strategies (2 symbols * 1 timeframe * 16 instances = 32). This is, of course, much less than the expected total number of instances in the final EA, but during optimization, the absolute majority of our passes will use either one instance (at the first stage) or no more than 16 instances (at the second stage). Therefore, such a test subject EA will suit us perfectly.
Let’s launch the EA in profiling mode on historical data. When running in this mode, a special version of the EA for profiling will be automatically compiled and launched in the strategy tester. Let’s quote the description of using profiling from the Reference:
Fig. 1. Results of profiling the code of the studied EA
By default, the profiling results list shows large functions located at the top levels. But by clicking on the string with the function name, we can see a nested list of functions that were called from this one. This allows us to more accurately determine which sections of code took up the most CPU time.
In the first two strings, we expectedly saw the OnTick() handler, as well as the CVirtualAdvisor::Tick() handler called from it. Indeed, in addition to initialization, the EA spends most of its time handling incoming ticks. But the third and fourth strings of results raise reasonable questions.
Why do we have so many calls to the current symbol select method? Why is so much time spent on getting some integer properties of the symbol? Let’s figure it out.
By expanding the string corresponding to the CSymbolInfo::Name(string name) method call, we can track that almost all the time is spent calling it from the function of checking the need to close the virtual position.
This code was written quite a long time ago. At that moment, it was important to us that open virtual positions were correctly translated into real positions. Closing a virtual position was supposed to result in an immediate (or almost immediate) closure of some volume of real positions. Therefore, this check should be performed on every tick and for every open virtual position.
For self-sufficiency, we provided each CVirtualOrder class object with its CSymbolInfo class object instance, through which we requested all the necessary information about prices and specifications of the required trading instrument (symbol). Thus, for 16 instances of trading strategies using three virtual positions each, there will be 16*3 = 48 of them in the array of virtual positions. If the EA contains several hundred instances of trading strategies, and also uses a larger number of virtual positions, then the number of calls to the symbol selection method will increase many times over. But is it necessary?
When do we really need to call the symbol name selector method? Only if the virtual position symbol has changed. If it has not changed since the previous tick, then calling this symbol method is useless. The symbol can only change when opening a virtual position that either has not been opened before or was opened for a different symbol. This clearly does not happen on every tick, but much, much less frequently. Moreover, in the model strategy used, there is never a change of symbol for one virtual position, since one instance of the trading strategy works with a single symbol, which will be the symbol for all virtual positions of this instance of the strategy.
Then you can send the CSymbolInfo class objects to the trading strategy instance level, but this may also be redundant, since different trading strategy instances may use the same symbol. Therefore, we will take them even higher – to the global level. At this level, we only need to have the number of instances of the CSymbolInfo class objects equal to the number of different symbols used in the EA. Each CSymbolInfo instance will be created only when the EA needs to access the properties of a new symbol. Once created, a copy will be permanently assigned to a specific symbol.
Inspired by the following example from the book, we will create our own class CSymbolsMonitor. Unlike the example, we will not create a new class, which, although written much more neatly, will essentially repeat the functionality of an existing class in the standard library. Our class will act as a container for several objects of the CSymbolInfo class and ensure that a separate information object of the class is created for each symbol used.
To make it accessible from anywhere in the code, we will again use the Singleton design pattern in the implementation. The base of the class is formed by the m_symbols[] array storing the pointers to the CSymbolInfo class objects.
The implementation of the static method for creating a single instance of a class is similar to the implementations that have already been encountered earlier. The destructor will contain a loop for deleting created information objects.
The public tick handling method will provide periodic updates of symbol specification and quote information. The specification may not change at all over time, but just in case, we will provide for its update once a day. We will update quotes every minute, since we use the EA’s operating mode only for opening minute bars (for better repeatability of modeling results in the 1 minute OHLC mode and the every tick mode based on real ticks).
Finally, we add an overloaded indexing operator to get a pointer to the desired object given a symbol name. It is in this operator that the automatic creation of new information objects for symbols that have not previously been accessed through this operator will occur.
Save the received code in the SymbolsMonitor.mqh file of the current folder. Now comes the turn of the code that will use the created class.
In this class, we already have several objects that exist in a single copy and perform some specific tasks: a receiver of virtual position volumes, a risk manager, and a user information interface. Let’s add a symbol monitor object to them. More precisely, we will create a class field that will store a pointer to the symbol monitor object:
The creation of the symbol monitor object will be initiated when the constructor is called by calling the CSymbolsMonitor::Instance() static method similar to other objects mentioned earlier. We will add the deletion of this object in the destructor.
Add calling the Tick() method to the new tick handler in order to monitor symbols. It is here that the quotes of all symbols used in the EA will be updated:
Taking this opportunity, let’s add the ChartEvent event handler to this class with an eye to the future. For now, the same-name method of the m_interface interface object will be called in it. It does nothing at this stage.
As already mentioned, obtaining information about symbols is performed in the class of virtual positions. Therefore, let’s start making changes from this class, and first of all, let’s add pointers to the monitor (CSymbolsMonitor class) and the information object for a symbol (CSymbolInfo class):
Adding pointers to the composition of class fields implies that they should be assigned pointers to some created objects. And if these objects are created inside the methods of objects of this class, then it is necessary to take care of their correct deletion.
Let’s add the initialization of the pointer to the symbol monitor and the clearing of the pointer to the symbol information object. Call the CSymbolsMonitor::Instance() static method to get the pointer to the symbol monitor. The creation of a single monitor object (if it does not exist) will occur inside it. In the destructor, add the deletion of the information object if it was created and has not yet been deleted:
I did not add receiving the pointer to the m_symbolInfo info object to the constructor since at the moment of calling the constructor it may not always be known exactly which symbol will be used in this virtual position. This becomes clear only when opening a virtual position, that is, when calling the CVirtualOrder::Open() method. We will add the initialization of the pointer to the symbol information object to it:
Now, since the symbol monitor is responsible for updating the symbol quotes information, we are now able to free the CVirtualOrder class from all calls of the Name() and RefreshRates() methods for the m_symbolInfo information object of symbol properties. When opening a virtual position in m_symbolInfo, we will save the pointer to the object the required symbol has already been selected for. When accompanying a previously opened virtual position, the RefreshRates() method was already called once on this tick — this was done by the symbol monitor for all of them in the CSymbolsMonitor::Tick() method.
Let’s do the profiling again. The picture has changed for the better, but calling the SymbolInfoDouble() function still occupies 9%. A quick search revealed that these calls are needed to obtain the spread value. But we can replace this operation with calculating the difference in prices (Ask — Bid), which have already been obtained when calling the RefreshRates() method and do not require additional SymbolInfoDouble() function calls.
Additionally, changes were made to this class that were not directly related to increasing the speed of operation and were not necessary for the model strategy under consideration:
Perhaps, this library is in for a more radical overhaul. Therefore, we will not dwell on the description of these changes.
To use the symbol monitor, we needed to make some minor edits to the trading strategy class as well. First, as in the class for virtual positions, we made it so that a member of the m_symbolInfo class now stores a pointer to the object instead of the object itself:
And added its initialization in the constructor:
We commented out the registration of the new bar event handler, since it will now be registered in the symbol monitor.
Secondly, we removed the update of the current prices from the strategy code (in the methods for checking the signal for opening and the opening of positions itself), since the symbol monitor also takes care of this.
Let’s compare the results of testing the EA under study on the same time interval before and after making changes related to the addition of the symbol monitor.
Fig. 2. Comparing test results of the previous version and the current one with the symbol monitor
As we can see, they are generally the same, but there are some minor differences. Let’s show them in the form of a table for clarity.
If we compare the first trades in the reports, we can see that the previous version features additional positions that are not present in the current one and vice versa. Most likely, this is due to the fact that when the tester is launched on the EURGBP symbol, a new bar for EURGBP occurs at mm:00, and for another symbol, for example GBPUSD, it can occur either at mm:00 or mm:20.
To eliminate this effect, we will add an additional check for the occurrence of a new bar to the strategy:
After this modification, the results only improved. The current version showed the highest normalized profit:
So let’s leave the changes made and move on to creating a database filling script.
We will not create a script, but an EA, but it will behave like a script. All work will be performed in the initialization function, after which the EA will be unloaded on the first tick. This implementation will allow us to run it both on the chart and in the optimizer, if we want to get multiple runs with parameters changing within the specified limits.
Since this is the first implementation, we will not think too much in advance about which set of inputs will be more convenient, but we will try to make just a minimal working prototype. Here is the list of parameters we ended up with:
The name and version of the project are obvious, then there are two parameters, in which we will pass lists of symbols and timeframes, separated by semicolons. They will be used to obtain single instances of the trading strategy. For each symbol, all timeframes will be taken in turn. So if we specified three symbols and three timeframes in the default values, this would result in nine single instances being created.
Each single instance must go through a first stage of optimization, where the best combinations of parameters are selected specifically for it. More precisely, during the optimization we might try many combinations, from which we can then select a certain number of “good” ones.
This choice will already be made at the second stage of optimization. As a result, we will have a group of several “good” instances working on a certain symbol and timeframe. After repeating the second step for all symbol-timeframe combinations, we will have nine groups of single instances for each combination.
During the third step, we will combine these nine groups, obtaining and storing in the library an initialization string, which can be used to create an EA that includes all single instances from these groups.
Let us recall that the code responsible for the sequential execution of all the above stages has already been written and can work if the necessary “instructions” are generated in the database. Before this, we added them to the database manually. Now we want to transfer this routine procedure to the developed EA script.
The remaining two parameters of this EA allow us to set the start and end dates of the optimization interval. We will use them to simulate periodic re-optimization and see how long after re-optimization the final EA will trade with the same results as in the optimization interval.
With that said, the initialization function code might look something like this:
That is, we sequentially create an entry in the project table, then add stages to the project stage table, and then fill in the work and task tables for each job. At the end, we set the project status to Queued. Thanks to triggers in the database, all stages, jobs and tasks of the project will also move to the Queued status.
Let’s now look at the code from the created functions in more detail. The simplest of them is to create a project. It contains one SQL query to insert data and store the ID of the newly created record in the id_project global variable:
As a project description, we form a string from the start and end dates of the optimization interval. This will allow us to distinguish between projects for the same version of the trading strategy.
The function for creating stages will be a little longer: it will require three SQL queries to create three stages. Of course, there may be more stages, but for now we will limit ourselves to only the three that were mentioned a little earlier. After creating each stage, we also store their IDs in the id_stage1, id_stage2 and id_stage3 global variables.
For each stage we specify its name, the ID of the parent stage and the name of the EA for the stage. The remaining fields in the stage table will be mostly the same for different stages: optimization interval, initial deposit, and so on.
The main work falls on the function of creating jobs and tasks CreateJobs(). Each job will be related to one combination of symbol and timeframe. So, first we create arrays for all used symbols and timeframes listed in the inputs. For timeframes, I have added the StringToTimeframe() function, which converts the timeframe name from a string to a value of the ENUM_TIMEFRAMES type.
Then, in a double loop, we go through all combinations of symbols and timeframes and create three optimization tasks with a custom criterion.
This number of tasks is determined, on the one hand, by the fact that we have accumulated at least 10-20 thousand passes during optimization on one combination, and on the other hand, there would not be so many of them that the time taken by optimization would be too long. The custom criterion for all three tasks is chosen because, with different runs, the genetic algorithm for this trading strategy almost always converges to different combinations of parameters. Therefore, there is no need to use different criteria for different runs, we already have a fairly rich choice of different good combinations of parameters for a single instance of the strategy.
In the future, the number of tasks and the optimization criteria used can be included in the script parameters, but now they are simply hard-coded in the code.
For each job of the first stage, we use the same optimization parameter template, which is specified in the paramsTemplate1 global variable :
Save the IDs of the added jobs to the id_jobs1 array for use in creating the second stage jobs.
To create the second stage works, the template specified in the paramsTemplate2 global variable is also used, but it already has a variable part:
The value that comes after “idParentJob_=” is the ID of the first stage job that uses a specific symbol and timeframe combination. Before the creation of the first stage jobs, these values are unknown, so they will be substituted into this template immediately before the creation of each second stage job from the id_jobs1 array.
The count_ parameter in this template is equal to 8, that is, we will collect groups of eight single instances of trading strategies. Our second stage EA allows us to set a value from 1 to 16 in this parameter. I chose the value 8 for the same reasons as the number of tasks for one job in the first stage – not too few and not too much. I might move it into the script inputs later.
At the second stage, we create only one optimization task for a single job, since in one optimization loop we select quite good groups of single instances of the trading strategy. We will use a user criterion as an optimization one.
We also save the IDs of the added jobs to the id_jobs2 array (we did not need them in the end). These IDs may be useful when adding stages, so we will not remove them.
At the third stage, the parameter template contains only the name of the final group, under which it will be added to the library:
We form the name of the final group from the name and version of the project, as well as from the end date of the optimization interval and substitute it into the template used to create the work of the third stage. Since at the third stage we sort of collect together the results of all the previous stages, then only one job and its task are created:
After this, all that remains is to change the project status so that it is queued for execution:
Save the changes made to the CreateProject.mq5 new file in the current folder.
There is one more thing. It is probably safe to assume that the database structure will be permanent, so it can be integrated into the library. To fulfill this task, we created the db.schema.sql file with the database structure as a set of SQL commands and connected it as a resource to Database.mqh:
We also slightly changed the logic of the Connect() method – if there is no database with the specified name, it will be automatically created using SQL commands from a file loaded as a resource. At the same time, we got rid of the ExecuteFile() method, since it is no longer used anywhere.
Finally, we have come to the point where we can try to run the implemented code.
We will not generate many projects at once, but will limit ourselves to only four. To do this, we will simply place the EA-script on any chart four times, setting the necessary parameters each time. Let the values of all parameters except the end date remain equal to the default values. We will change the end date by adding an additional month to the test interval every time.
As a result, we get approximately the following database content. The project table features four projects:
The stage table has four stages per each project. An additional stage named “Single tester pass” is created automatically when creating the project and used when we want to launch a single strategy tester pass outside of the auto optimization conveyor:
The corresponding jobs have been added to the job table:
After the projects were launched for execution, the result was obtained in approximately four days. This is certainly not such a small amount of time, despite efforts to optimize performance. But not so big either so that it cannot be allocated. We can see it in the strategy_groups group library table:
Or we can substitute the pass ID as an input of the SimpleVolumesStage3.ex5 third stage EA and run it in the tester at the selected time interval:
We will stop here for now and conduct a more detailed analysis of the results obtained in the coming articles.
So, we got the ability to automatically create tasks to launch the auto optimization conveyor, which includes three stages. This is still nothing more than a draft that will allow us to identify preferred directions for further development. The issues of implementing auto merging or replacing the initialization strings of the final EAs upon completion of the conveyor stages for each project remain open.
But one thing can already be said for sure. The chosen order of execution of optimization tasks in the conveyor is not very good. Now we have to wait for the full completion of all the work of the first stage in order to begin the second. And in the same way, the third stage will not begin until all the work of the second stage is completed. If we plan to somehow implement a “hot” replacement of the initialization strings of the final EA, which continuously works on the account in parallel with the optimization being carried out, then we can make these updates smaller, but more frequent. This may improve the results, but it is still only a hypothesis that needs to be tested.
It is also worth noting that the developed EA-script is focused on creating optimization projects only for the considered model trading strategy. Another strategy will require some minor changes to the source code. At a minimum, you will have to change the template of the input parameter string for the first stage of optimization. We have not yet moved these templates into inputs, since it is inconvenient to set them there directly. However, further on, we will probably develop some format for describing the task for creating a project, which the script EA will upload from a file.
