We resume our work we started in the previous article. Let us remind you that after dividing the entire project code into the library and project parts, we decided to check how we can move on from the SimpleVolumes model trading strategy to another one. What do we need to do for this? How easy will it be? It goes without saying that it was necessary to write a class for a new trading strategy. But then some unobvious complications arose.
They were connected precisely with the desire to ensure that the library part could be independent from the project part. If we had decided to break this newly introduced rule, there would have been no difficulty. However, a way was eventually found to both preserve code separation and enable the integration of the new trading strategy. This required changes to the library files of the project, although not very large in volume, but significant in meaning.
As a result, we were able to compile and run the optimization of the first stage EA with a new strategy called SimpleCandles. The next steps were to get it working with the auto optimization conveyor. For the previous strategy, we developed the CreateProject.mq5 EA, which made it possible to create a task optimization database for execution on the conveyor. In the EA parameters, we could specify which trading instruments (symbols) and timeframes we wanted to optimize, the names of the EA stages, and other necessary information. If the optimization database did not exist before, it was created automatically.
Let’s see how to make it work with the new trading strategy now.
We will start the main work by analyzing the CreateProject.mq5 EA code. Our goal will be to identify code that is the same, or nearly the same, across different projects. This code can be separated into a library section, splitting it into several separate files if necessary. We will leave the part of the code that will be different for different projects in the project section and describe what changes will need to be made to it.
But first, let’s fix a discovered error that occurs when saving tester pass information to the optimization database, refine the macros for organizing cycles, and look at how to add new parameters to a previously developed trading strategy.
In recent articles, we have started using relatively short testing intervals for optimization projects. Instead of intervals lasting 5 years or more, we began to take intervals lasting several months. This was due to the fact that our main task was to test the operation of the auto optimization conveyor mechanism, and reducing the interval allowed us to significantly reduce the time of an individual test pass, and therefore the overall optimization time.
To save information about passes to the optimization database, each test agent (local, remote, or cloud) sends it as part of a data frame to the terminal where the optimization process is running. In this terminal, after the optimization starts, an additional instance of the optimized EA is launched in a special mode – the data frame collection mode. This instance is not launched in the tester, but on a separate terminal chart. It will receive and save all information coming from test agents.
Although the code for the event handler for the arrival of new dataframes from test agents does not contain asynchronous operations, during optimization, messages about database insertion errors related to the database being locked by another operation began to appear. This error was relatively rare. However, several dozens out of several thousands runs ultimately failed to add their results to the optimization database.
It appears that the cause of these errors is the increasing number of situations where multiple test agents simultaneously complete a run and send a dataframe to the EA in the main terminal. And this EA tries to insert a new entry into the database faster than the previous insert operation can be completed on the database side.
To fix this, we will add a separate handler for this category of errors. If the cause of the error is precisely the database or table being locked by another operation, then we simply need to repeat the unsuccessful operation after some time. If after a certain number of attempts to reinsert data, the same error occurs again, then attempts should be stopped.
For insertion, we use the CDatabase::ExecuteTransaction() method, so let’s make the following changes to it. Add the request execution attempt counter to the method arguments. If an error of this kind occurs, pause for a random number of milliseconds (0 – 50) and call the same function with an increased attempt counter value.
Just in case, let’s make the same changes to the CDatabase::Execute() method for executing an SQL query without a transaction.
Another small change that will be useful to us in the future was to add a static boolean variable to the CDatabase class. It will remember that an error occurred while executing requests:
Let’s mention one change that has been long overdue. As you might remember, we created the FOREACH(A, D) macro to simplify the writing of the headers of loops that should iterate over all the values in a certain array:
Here Ais an array name, while Dis a loop body. This implementation had a drawback in that it was impossible to properly track the step-by-step execution of the code inside the loop body when debugging. Although this was rarely required, it was very inconvenient. One day, while browsing the documentation, I saw another way to implement a similar macro. The macro only specified the loop header, and the body was moved outside the macro. However, there was one more parameter that specified the name of the loop variable.
In our previous implementation, the name of the loop variable (the array element index) was fixed (i), and this did not cause any problems anywhere. Even in the place where a double loop was needed, it was possible to get by with the same names due to the different scopes of these indices. Therefore, the new implementation also received a fixed index name. The only parameter passed is the name of the array to be iterated over in the loop:
To switch to the new version, it was necessary to make changes in all places where this macro was used. For example:
Along with this macro, we added another one that provides the creation of a loop header. In the macro, each element of the A array is placed into the E array (which should be announced in advance) one by one. Before the loop header, the first element of the array, if it exists, is placed into this variable. As a loop variable we will use a variable with a name consisting of the i letter and E variable name. In the third part of the loop header, we increment the loop variable, while the E variable receives the value of the A array element with an increased index. Taking an index by modulo of the number of array elements allows us to avoid going beyond the array bounds on the last iteration of the loop:
Like almost all the code, the implementation of a trading strategy is also subject to change. If these changes concern the change in the composition of the input parameters of a single instance of a trading strategy, then it will be necessary to make edits not only to the trading strategy class, but also to some other places. Let’s look at an example to see what needs to be done for this.
Let’s assume that we decide to add a maximum spread parameter to the trading strategy. Its use will consist in the fact that if at the moment of receiving a signal to open a position the current spread exceeds the value set in this parameter, then the position will not open.
To begin with, we will add an input to the first stage EA, through which we can set this value when running the tester. Then, in the initialization string forming function, add substitution of the new parameter value to the initialization string:
The initialization string now contains one more parameter than before. So the next change will be to add the new property of the class and read the values from the initialization string in the constructor into it:
Now the new parameter can be used as we wish in the methods of the trading strategy class. Based on its purpose, the following code can be added to the position open signal receiving method.
Similarly, we can add other new parameters to trading strategies or get rid of parameters that have become unnecessary.
Let’s start analyzing the CreateProject.mq5 project creation EA code. In its initialization function, we have already split the code into separate functions. The purpose of each is clear from the name:
This division is not very convenient, because the selected functions turned out to be quite cumbersome and solve quite different problems. For example, in the CreateJobs() function, we pre-process input data, generate parameter templates for jobs, insert information into the database, and then perform similar actions to create optimization tasks in the database. It would be better if it were the other way around: the functions were simpler and solved one small problem.
To use the new strategy in the current implementation, we would need to change the template of the first stage parameters, and possibly also the number of tasks with optimization criteria for it. The first stage parameter template for the previous trading strategy was specified in the paramsTemplate1 global variable :
Fortunately, it was the same for all first stage optimization jobs. But this may not always be the case. For example, in the new strategy, we included the symbol values and the timeframe the strategy should work on into the parameters. This means that in different first stage optimization jobs created for different symbols and timeframes, the parameters template will have variable parts. However, to set their values, you will need to delve into the depths of the task creation function code and make changes to it. Then it will no longer be possible to take it to the library section.
In addition, our optimization project creation EA now creates a project with three fixed stages. We arrived at this simple set of stages during the development, although we tried adding other stages (see for example, part 18 and part 19). Additional steps did not show any significant improvement in the final result, although this may not be the case for other trading strategies. Therefore, if we move the current code into the library part, we will not be able to change the composition of the stages in the future, if we wish.
So, as much as we would like to get by with a little effort, it is still better to do some serious refactoring work on this code now than to put it off until later. Let’s try to split the project creation EA code into several classes. The classes will be moved to the library section, and in the project section we will use them to create projects with the desired composition of stages and their content. At the same time, this will also serve as a template for the future display of information about the conveyor progress.
To begin, we tried to write what the final code might look like. This preliminary version remained virtually unchanged until the final version was released. Only specific parameter compositions have been added to method calls. Therefore, let’s see what the new version of the initialization function for the optimization project creation EA looks like. To avoid distraction by small details, the arguments of the methods are not shown:
With this code structure, we can easily add new stages and flexibly change their parameters. But for now we only see one new class that we will definitely need — the COptimizationProject optimization project class. Let’s look at its code.
While developing this class, it quickly became clear that we would need separate classes for all the types of entities that we store in the optimization database. So next come the COptimizationStage classes for the project stages, COptimizationJob for the project stage jobs and COptimizationTask for the tasks of each project stage job.
Since the objects of these classes are, in essence, a representation of entries from various tables of the optimization database, the composition of the class fields will repeat the composition of the fields of the corresponding tables. In addition to these fields, we will add other fields and methods to these classes that are necessary to perform the tasks assigned to them.
For now, we will make all properties and methods of the created classes public for simplicity. Each class will have its own method for creating a new entry in the optimization database. In the future, we will add methods for changing an existing entry and reading an entry from the database, since we will not need it when creating the project.
Instead of the previously used tester parameter templates, we will create separate functions that will return already filled parameters according to the template. This way the parameter templates will move inside these functions. These functions will take a project pointer as a parameter and will be able to use it to access the required project information to be substituted into the template. We will move the declaration of these functions to the project section, and in the library section we will declare only a new type – a pointer to the function of the following type:
Thanks to this, we will be able to use the stages parameters generation functions in the COptimizationProject class. They do not exist yet, but in the future, in the design part, we will definitely have to add them.
Here is what the description of this class looks like:
At the beginning are the properties that are directly stored in the optimization database in the projects table. Next come arrays of all project stages, jobs and tasks, and then properties for the current state of the project creation.
Since this class currently has only one task (creating a project in the optimization database), we immediately connect to the required database in the constructor and open a transaction. The completion of this transaction will occur in the destructor. This is where the CDatabase::s_res static class field comes in handy. Its value can be used to determine whether any error occurred when inserting entries into the optimization database when creating a project. If there were no errors, the transaction is confirmed, otherwise it is canceled. Also, memory for created dynamic objects is freed in the destructor.
The methods for adding jobs and tasks are declared in two variants. In the first one, the lists of symbols, timeframes and criteria are passed to them in string parameters, separated by commas. Inside the method, these strings are converted into arrays of values and substituted as arguments when calling the second version of the method, which accepts arrays.
The third argument is the pointer to the function for creating optimization parameters for stage EAs.
This class description has many properties compared to other classes, but this is only due to the fact that there are multiple fields in the stages table of the optimization database. For each of them, there is a corresponding property in this class. Also note that the pointer to the project object (which includes this stage) and the pointer to the previous stage object are passed to the stage constructor. For the first stage there is no previous one, so we will pass NULL for it in this parameter.
The actions performed in the constructor and in the method of inserting a new entry into the stages table is very simple: remember the passed argument values in the object properties and use them to form an SQL query to insert an entry into the desired optimization database table.
This class is identical in structure to the COptimizationStage class. The constructor remembers the parameters, while the Insert() method inserts a new row into the jobs table in the optimization database. Also, the pointer to the stage object (which is to include the current job object) is passed to each job object during creation.
The last remaining COptimizationTask class is constructed in the same way, so I will not provide its code here.
Let’s return to the CreateProject.mq5 file and look at its main parameters. This file is located in the project section, so for each individual project we can specify the required default parameter values in it so as not to change them at startup.
First of all, we specify the name of the optimization database:
In the next group of parameters, we specify the comma-separated symbols and timeframes the first and second stages of the EA optimization will be performed on:
With this selection, six jobs will be created for each of the possible combinations of three symbols and two timeframes.
Next comes the selection of the interval, over which optimization will take place:
In the account parameters group, we select the main symbol that will be used in the third stage, when the EA will work with several symbols in the tester. Its selection becomes important if among the symbols there are those, for which trading continues on weekends (for example, crypto currencies). In this case, we need to select this one as the main one, since otherwise, during the tester run, it will not generate ticks on all weekends.
In the first stage parameters group, the name of the first stage EA is specified, although it may remain the same. Next, we specify the optimization criteria that will be used for each job in the first stage. These are just numbers separated by commas. The value of 6 corresponds to the user optimization criterion.
In this case, we specified the user criterion three times, so each job will contain three optimization problems with the specified criterion.
In the second stage parameters group, we have added the ability to specify all the values of the second stage EA parameters, and not just the name and number of strategies in the group. These parameters influence the selection of the first stage passes, whose parameters will be used to select groups in the second stage.
For example, if stage2MinTrades_ =20 only those individual trading strategy instances that completed at least 20 trades in the first stage will be able to join the group. The stage2UseClusters_ parameter has been commented out for now as we are not currently using clustering of the second stage results. Therefore, it should be substituted with false.
We also added some things to the third stage parameters group. In addition to the name of the third stage EA (which also does not need to be changed when changing projects), two parameters have been added that control the formation of the name of the final EA’s database. In the final EA itself, this name is formed in the CVirtualAdvisor::FileName() function according to the following template:
Therefore, the third stage EA uses the same template. is replaced with projectName_, while with stage3Magic_. The stage3Tester_ parameter is responsible for adding the “.test” suffix.
In principle, it would be possible to create one parameter that would simply indicate the full name of the final EA database. After completing the third stage, the resulting file of this database can be safely renamed as desired before further use.
Now we just need to create functions for generating parameters for stage EAs using given templates. Since we are using three stages, we will need three functions.
For the first stage, the function will look like this:
It is based on the optimization parameters of the first stage EA copied from the strategy tester with the desired ranges for iterating over individual input parameters set. This string is filled with the values of the symbol and timeframe, for which a job object is created in the project at the time this function is called. For example, if for a certain timeframe it is necessary to use other ranges of inputs to be iterated over, then this logic can be implemented in this function.
When moving to another project with a different trading strategy, this function should be replaced with another one, written for the new trading strategy and its set of inputs.
For the second and third stages, we also implemented these functions in the CreateProject.mq5 file. However, when moving to another project, they most likely will not have to be changed. But let’s not take them to the library section right away. Let them stay here for now:
Next comes the code for the initialization function, which does all the work and removes the EA from the chart before finishing. Let’s show it now with the parameters of the called functions:
This part of the code also does not need to be changed when moving to another project, unless we want to change the composition of the auto optimization conveyor stages. Over time, we will improve it, too. For example, the code currently contains numeric constants that should be replaced with named constants for better readability. If it turns out that this code really does not need any changes, then we will move it to the library section.
So, the EA for creating optimization projects in the database is ready. Now let’s create stage EAs.
We have already implemented Stage1.mq5 in the previous article, so now we have made changes to it related only to the addition of the new maxSpread_ parameter into the trading strategy. These changes have already been discussed above.
In the second and third stage EAs, we only need to define the __NAME__ constant with the unique EA name and connect the file or files of the trading strategies used. The rest of the code will be taken from the included library file of the corresponding stage. Here is what the code for the second stage EA might look like Stage2.mq5:
In the final EA, we only need to add the connection to the strategy used. We do not need to declare the __NAME__ constant here, since in this case both the constant and the function for generating the initialization string will be declared in the included file from the library part. In the code below, we have shown in the comments what the EA name and the function for generating the initialization string look like in this case:
If we suddenly want to change something from this, then it is enough to remove the comments from this code and make the necessary edits to it.
Thus, in the project part we will have the following files:
Let’s compile all the files of the project part so that for each file with the extension of mq5 a file with the extension of ex5 is created.
Drag the CreateProject.ex5 EA to any chart in the terminal (this EA does not need to be run in the tester!). In the EA source code, we have already tried to specify the current values for all inputs, so you can simply click OK in the dialog.
Fig. 1. Launching the project creation EA in the optimization database
Drag the Optimization.ex5 EA (this EA also does not need to be run in the tester!) to any chart. In the dialog that opens, enable the use of the DLL, and ensure that we have specified the correct optimization database name:
If all is well, we should see something like this: in the tester, the optimization of the first stage EA starts on the first symbol-timeframe pair, while we will see the following on the chart with the Optimization.ex5 EA: “Total tasks in queue: …, Current Task ID: …”.
Next, you should wait for some time until all optimization tasks are completed. This time can be quite significant if the testing interval is long and the number of symbols and timeframes is large. With the current default settings on 33 agents, the entire process took about four hours.
At the last stage of the conveyor, optimization is no longer performed, but a single pass of the third stage EA is launched. As a result, a file with the database of the final EA is created. Since we chose the project name “SimpleCandles” when creating the project, while the magic number is 27183 and stage3Tester_=true, then a file named SimpleCandles-27183.test.db.sqlite will be created in the shared terminal.
Step 3: Launching the final EA in the tester
Let’s try running the final EA in the tester. Since its code is now completely taken from the library part, the default parameter values are defined there as well. Therefore, when we launch the SimpleCandles.ex5 EA in the tester without changing the values of the inputs, it will use the last added strategy group (groupId_= 0) with auto updates enabled (useAutoUpdate_= true) from the database named SimpleCandles-27183.test.db.sqlite (SimpleCandles EA file name plus the default magic number magic_= 27183 and plus “.test” suffix due to running in the tester).
Unfortunately, we have not yet created any special tools that allow us to view existing strategy group IDs in the final EA’s database. We can only open the database itself in any SQLite editor and view them in the strategy_groups table.
However, if only one optimization project was created and run once, then only one strategy group with ID 1 will appear in the final EA database. Therefore, it makes no difference whether we specify a specific groupId_= 1 or leave groupId_= 0 from the point of view of group selection. In any case, the only existing group will be loaded. If we run the same project again (this can be done by changing the project status directly in the database) or create another similar one and run it, then new strategy groups will appear in the final EA’s database. In this case, different groups will be used for different groupId_ parameter values.
Auto update enable parameter (useAutoUpdate_= true) also requires our attention. Even though there is only one group, this parameter affects the operation of the final EA. This is manifested in the fact that when auto update is enabled, only those strategy groups whose appearance date is less than the current simulated date can be loaded for work.
This means that if we run the final advisor on the same interval that we used for optimization (2022.09.01 – 2023.01.01), then our only strategy group will not be loaded, since it has the formation date of 2023.01.01. Therefore, we need to either turn off auto updates (useAutoUpdate_= false) and specify the specific ID of the trading strategy group used (groupId_= 1) in the inputs when launching the final EA, or select another interval located after the end date of the optimization interval.
In general, until we have finally chosen which strategies will be used in the final EA and have not set the goal of testing them for the feasibility of periodic re-optimization, this parameter can be set to false and specify the specific ID of the trading strategy group being used.
The last set of important parameters is responsible for what database name the final EA will use. In its default settings, the magic number is the same as the one we specified in the settings when creating the project. We also made the name of the final EA file match the name of the project. When creating the project, the stage3Tester_ parameter value was equal to true, so the file name of the created database of the final EA will be SimpleCandles-27183.test.db.sqlite. It completely matches the one the final SimpleCandles.ex5 EA will use.
Let’s look at the results of running the final EA on the optimization interval:
Fig. 4. The auto optimization EA operation on the interval 2022.09.01 – 2023.01.01
If we run it on some other time interval, the results will most likely not be as pretty:
Fig. 5. The auto optimization EA operation on the interval 2023.01.01 – 2023.02.01
We took the interval of one month immediately after the optimization interval as an example. Indeed, the drawdown slightly exceeded the expected value of 10%, and the normalized profit decreased by about five times. Is it possible to re-run the optimization for the last three months and get a similar picture of the EA’s behavior over the next month? This question remains open for now.
Step 4: Launching the final EA on a trading account
To run the final EA on a trading account, we will need to adjust the name of the resulting database file. We should remove the “.test” suffix from it. In other words, we simply rename and copy SimpleCandles-27183.test.db.sqlite to SimpleCandles-27183.db.sqlite. Its location remains the same – in the common terminal folder.
Drag and drop the final SimpleCandles.ex5 EA to any terminal chart. In the inputs, we might leave everything with the default values, since we are quite satisfied with loading the last group of strategies, and the current date will obviously be greater than the creation date of this group.
While preparing the article, the finalized EA was tested on a demo account for about a week and showed the following results:
Fig. 7. Results of the final EA operation on the trading account
It was a pretty good week for the EA. With the drawdown of 1.27%, the profit was about 2%. The EA restarted a couple of times due to a computer reboot, but successfully restored information about open virtual positions and continued working.
Let’s see what we got. We have finally put together the results of a fairly lengthy development process into something that resembles a coherent system. The resulting tool for organizing auto optimization and testing of trading strategies allows for significant improvements in the testing results of even simple trading strategies through diversification across different trading instruments.
It also allows for a significant reduction in the number of operations that require manual intervention to achieve the same goals. Now there is no need to track the completion of yet another optimization before launching the next one, no need to think about how to save intermediate optimization results and how to then integrate them into a trading EA. Instead, we can focus directly on developing the logic behind our trading strategies.
Of course, there is still a lot that can be done to improve and make this tool more convenient. The idea of a fully-fledged web interface that manages not only the creation, launch, and monitoring of running optimization projects, but also the operation of EAs running in various terminals and viewing their statistics remains in the distant future. This is a very large task, but, looking back, the same can be said about the task that today has already received a more or less complete solution.
