Automated migration procedures

Introduction

The presented procedure offers the following characteristics:

• It is divided as follows:
• Distinct steps are defined with a mandatory sequential running order (Initialization first, then Common data, then Modules, and finally Postmigration). The Modules steps are linked to a functional module,
• at each step, phases numbered 1 to 9 are defined. In these phases, unitary migration operations, otherwise named procedures, are listed (each procedure is given an order number).
• procedures are run in a sequential order: procedures within phase N in step P can only be launched if all the procedures linked to phases 1 to N-1 of step P, as well as all the procedures linked to steps 1 to P-1, if they exist, have been completed without any error,
• the procedures linked to a same step and phase can be run in parallel. They have a unique rank that defines the launching order (this information can be modified), but according to the launching parameters, various tasks in a same step and phase can be run simultaneously.

• It is used to add specific procedures and to easily sequence them. A technical annex describes how to develop new procedures.
• Its operation is based on data copying rather than data updating. It provides more efficiency in database operations:
• by creating a destination table that will not be indexed before the copy is over, which is much more efficient.
• by using fast grouped insertions,
• by offering the possibility to restart a procedure, whenever there is an execution error, by deleting the data that is already transcoded,
• by offering the possibility to temporarily interrupt a procedure, resumption marks being available to process the remaining data only.
• by offering the possibility to postpone, if necessary, the copy of old data, the absence of which will not affect an initial reuse in downgraded mode.
• It is based on a sequence monitor, which is used to view the phases and their execution.

Main steps of the migration

The phases to comply with, in case of migration of a folder from one major release to another, are the following:

• running the premigration controls in the original folder. These controls can be carried out without stopping the ERP operations.
• Installing version 7 in a dedicated environment.
• The extraction of the data to be migrated from the original folder, followed by the reintegration of this data (operations must then be stopped and started again when migration is completed). The extraction of data will be carried in a "flat" format for small folders, and by using database extraction tools for larger folders.
• Revalidating the folder in the V7 environment. This revalidation can be divided into several steps using the migration plan setup. If the folder is not too bulky, defining migration plans may not prove necessary. It becomes necessary if transfer processing routines need to be run in parallel (to benefit from the power of a multi-processor server) and their sequencing needs to be checked using resumption marks.
• Postmigration controls. This mainly relates to setup checking and adaptation.

The rest of the documentation describes all the steps.

Premigration steps

Premigration steps are launched in the source environment and they can be spread over a period of time, before the actual migration. It is not necessary to stop any operations.

Premigration steps are divided as follows:

• first, it is necessary to check that the original folder is at the minimum patch level required to run a migration. This level, which can be displayed via the ? menu > About, must be at least:
• patch 10 for a version 5,
• patch 21 for a version 6,
• then, the functional prerequisites necessary for the migration processing must be checked. These prerequisites are described in the dedicated documentation.
• Then, automated procedures can be triggered in the original folder, via processing routines with names beginning with UU. These automated procedures can be launched using the dedicated function. These processing routines are supplied upon request, according to the release number, as a specific patch list dedicated to migration. Two types of procedures are delivered:
• first, control procedures of the data contained in the original folder. The result of this control can reveal errors or inconsistencies that it is imperative to correct before launching the migration proper. This control phase can be anticipated several weeks ahead of the migration, in order to have enough time to correct the data. It will nevertheless be necessary to relaunch it to check that everything is correct just before the transition to the next version. These procedures are listed in the corresponding document.
• then, optional loading procedures, that revalidate some tables by adding additional fields, then by populating them. Since these procedures are independent from one another, they can be planned over several weeks prior to migration (operations can resume as soon as the addition of fields is completed). These procedures aim at significantly accelerating the actual migration steps. However, not launching these procedures does not jeopardize the execution of the migration procedures. These procedures are listed in the corresponding document.

Migration plan

This environment is prepared by installing solution V7, and then integrating all the existing patches in its X3 folder.
 A V7 solution cannot and must not be installed on a former solution.

Operation process in a migration plan

This step is the beginning of the transfer. At this stage, the operational use of the solution must be stopped: it will resume when the migration is complete.

This step is sub-divided in 4 phases:

• extraction of the data from the folder to be migrated,
• data is extracted in a "flat" format for small folders. The extraction function is used for that purpose. It creates the files in the default SVG directory.
• databases are extracted as soon as the volume is larger (the format depends on the database and tool used),
• copy of the files of the folder to be migrated to the V7 solution,
• a copy folder of the folder tree structure is created in the new environment of version 6, and the exported data is then imported into the new environment,
• the folder record is created in the V7 environment. If the import is carried out via the console, the folder record is created automatically. On the other hand, if the import has been carried out via database tools, the folder record will have to be created manually after the import is completed.

 You can group these various steps into a single one.  For that purpose, you can use theremote import wizard available in the SAFE X3 configuration console.

If need be, activity code values can be modified in the folder record, to implement, for instance, new functionalities controlled by new activity codes.The modification of the folder record, or its creation, will be carried out by logging onto the "Supervisor" folder (X3 in the case of Sage X3).

Final purge of the entry tables

Revalidating the folder transforms the structure of the X3 dictionary, then of the setup tables, then the data tables, of the previous version in order to bring the structure up to the V7 level while transferring the data. It is triggered via the Validation button from the folder management.

It performs the supervisor and functional migration of the folder, by running a comparison between the dictionary of the new version and the dictionary of the version to migrate.

Main revalidation steps

The folder revalidation triggers the following steps in the folder to migrate:

• to purge some temporary or totals tables which recreation will be carried out in the course of future steps. The supervisor tables impacted by this purging are the following:
      ALSTRD
      ALISTER
      AWRKLOG
      AWRKLOGIND
      AWRKLOGMES
      ATMPTRA
  Other temporary functional tables are also purged (in the case of Sage X3, these tables are: LABELPRN, PRTSCRWRK, BALANCE, BALANA, GLCONSO, BALCONSO, GAJOUSTA, FUP, FUPTOT, BATCH, TMPEXPENSE).
• migration of the envelope, that is to say, the database dictionary and the supervisor tables necessary for the operating of the database (connection, user management, development environment and minimum setup). Once this migration is carried out, it is theoretically possible to connect, even though the tables containing the flow application data are still empty.
• functional migration: whenever there is a change in the structure of a table, the process will carry out the necessary changes, eventually by assigning appropriate default values through procedures that are sequenced in steps and phases. This is the lengthy part of the migration, the part that depends on the volume of data to migrate.
• postmigration process (they are essentially resynchronizations of the totals tables). Some of these steps can be postponed and the operations can resume but the software will be functioning in a downgraded mode during the intermediate phase. To consider the migration as completed, these steps must be carried out.
• final deletion of temporary tables. This is a manual phase. It can be postponed, for safety reasons, as long as there is no absolute certainty that all the the flow data has been correctly migrated.

Use of migration plans

The functional migration can be lengthy when the folder is very large. Moreover, some updated tables are potentially independent and, therefore, could be migrated in parallel, thus benefiting from multi-processor architectures to accelerate this phase.

To allow this migration to be scheduled in the best way, it is possible to define a customized migration plan before launching the data validation. This migration plan is described in the migration procedure function (called from the Supervisor folder). This function helps define the steps, phases and procedures of the functional and supervisor migration. A migration plan corresponds to the definition of the specific migration parameters (impacted folder, number of procedures that can be run in parallel, task sequencing policy, running status). A migration plan is created by recopying the whole of the active elements of the migration procedure. This complete set of procedures is provided as standard, allowing the entirety of the processing routines run by the standard software during the migration phase to be sequenced. A short description of the standard procedures is provided in a dedicated document.

It is possible, at this stage, to define specific procedures by writing complementary processing routines in conformity with the methodology defined in the following annex. These specific procedures will be inserted between the standard procedures in a relevant way.

Once a migration plan has been defined,it is possible to launch it, interrupt it momentarily, resume its running, and view its progress status.

When the volume of a folder is not too large and its migration does not require any particular planning, it is not necessary to create a migration plan. In effect, when there is no migration plan, the validation operation of the folder will automatically create a migration plan. The code of this migration plan will be the folder code, except if this code already corresponds to a migration plan that does not have the Pending status. If it were the case, the plan would be created with a predefined code under the form MIGmmddM##. mmandddare the numbers representing the month and the day of the launching, ## is a sequential number.

However, to customize the migration sequencing options, it is possible to create a migration plan with a code that must correspond to the folder name. This creation will take place in the supervisor folder before launching the folder validation. If a migration plan exists, with the Pending status, it will be used for the functional migration of the folder.

 A plan created with a name that is different from the folder to migrate will never be used by the automatic folder validation functions. Such a plan is restricted to a manual launching.

Operation process in a migration plan

A migration plan is characterized by a folder code and 4 parameters:

• a title,
• the number of procedures likely to be launched simultaneously. This number is a possible maximum. Only procedures that can be run in parallel will be launched simultaneously. Caution, this number is limited by the number of batch tasks likely to be run simultaneously (defined in the batch server setup function). This number is also limited by the user license,
• an indicator specifying whether the procedures must be launched automatically and follow the logic defined by the plan. If not, the sequencing of the procedures described in the plan will occur manually,
• an indicator stating if the postmigration procedures must be linked to the migration itself. These operations are detailed below.

If the migration plan is created by default upon folder revalidation, it is created with the following values:

• Number of simultaneous taskshas the value of the NBRTACSIM parameter or, if it does not exist, it has the value 2,
• Automatic sequencing has the value Yes,
• Postmigration operations has the value Yes.

It will always be possible for the user to modify these values from the migration plan control function.

The ordered list of the plan procedures is displayed in the screen associated with the migration plan. Specific buttons make it possible to globally control the running of the plan:

• by launching its execution,
• by momentarily interrupting the launching of new procedures,
• by interrupting all the procedures in progress,
• by relaunching its execution.

Each line of the plan materializes a step in the execution, characterized by:

• the code of the procedure and its title,
• the step and phase to which the procedure belongs
• the functional module to which the procedure is linked.
• the execution rank.

The migration steps make it possible to split the functional migration. If a procedure linked to a given phase is not completed, the following phases cannot be launched. The phases are linked to the following steps:

• the initialization step is a preliminary step that is the very first to be completed. It has not been used so far. It enables specific procedures to be executed before the migration itself,
• the common data step allows data used by the following steps to be migrated,
• the module step is used to to carry out migrations linked to each functional module. An operation belonging to this step is associated with a functional module: this association does not have any binding effect on the execution. For example, some data migrations linked to different modules can be run in parallel if they belong to the same phase. Their launching will follow the order of the modules, then of the execution ranks,
• the postmigration step is used to carry out synchronization operations or secondary updates whose execution is not mandatory to restart operations in a downgraded mode. It can be used, for example, to postpone the migration of old data. Please note nonetheless that even though this step is not mandatory to restart operations, it must be carried out later on for a folder to be considered as completely migrated. These postmigration operations are launched by default in a migration plan but they are optional. They are listed in a dedicated paragraphin the annex describing the migration procedures.

One step includes unitary procedures that are organized in phases and ranks.

• It is considered possible for two procedures defined in the same phase to be run simultaneously. As a consequence they must be independent. The assignment of a standard procedure to a given phase cannot be changed. As long as all the procedures belonging to a phase are not completed, the following phase cannot be launched.
• The rank order is purely a preferential launching order, but it can be modified, including for standard procedures, when the plan is defined.

Final purge of the entry tables

This phase is manual It is triggered by the execution of the TRTMIGDEL processing from the processing execution function. Running this phase will delete all the tables having the MIG activity code.

This phase cannot be reversed.Please ensure that the migration processing routines have been carried out successfully.

Keeping the temporary tables in the folder is compatible with the resumption of normal operations. It is therefore possible to keep these tables on-line for a few weeks of operation. Should any problem occur in the weeks following the migration, it will be possible to have the original data available for comparison or analysis purposes.

Example of sequencing on flow migration

Let us consider the following operations:

• in the initialization step, 7 operations:
• IN1, IN2 in phase 1 (ranks 3 and 7),
• IN3 and IN4, IN5 and IN6 in phase 2 (and successive ranks 1 3, 5 9),
• IN7 in phase 3,
• in the common data (TC) step, 6 operations:
• TC1, TC2, TC3, all 3 in phase 1, with ranks 3, 6, 9,
• TC4 in phase 2,
• TC5 and TC6 in phase 3,
• in the Module step, 5 operations:
• step MO1, in the Financials module, with phase 2 and rank 1,
• steps MO2 and MO3, in the Stock module, in phases 1 and 2 respectively, with ranks 1 and 2,
• steps MO4 and MO5, in the Sales module, with phases 1 and 2 respectively, with ranks 1,
• steps MO6 and MO7, in the Purchasing module, with phases 1 and 5 respectively, with ranks 1,

Let us assume that the migration plan is launched with a sequencing of tasks, and a maximum number of 2 procedures running simultaneously. The sequencing cab be the following:

• Procedures IN1 and IN2 are launched in this order,
• Let us assume that procedure IN2 is completed first. As long as procedure IN1 is not completed, no other procedure can be launched, because the following procedures are included in a phase or a step with a higher rank,
• when procedure IN1 is over, IN3 and IN4 procedures are launched in this order.
• If procedure IN4 is completed before procedure IN3, procedure IN5 will be launched. If it is completed before procedure IN3 is over, procedure IN6 will be launched and will be running at the same time as procedure IN3, which will continue to take place,
• when procedures IN3 and IN6 are both completed (and only then), the initialization step will be over.
• The procedures of the first phase of the Common Data step can then be launched: first of all, procedures TC1 and TC2, then TC3, which will be launched the moment that one of the two previous procedures have been completed. All three procedures TC1, TC2, TC3 must be completed before launching TC4, which will be running on its own since there are no other procedure in this phase. Then, TC5 and TC6 will be launched, and running in parallel.
• When procedures TC5 and TC6 are both completed, tasks belonging to the module step can be started,
• First, procedures MO2 and MO6 will be launched (in this order, these procedures run in parallel); then, as soon as one of these two procedures is completed, procedure MO4, the next one in the phase/module/rank order, will be launched in parallel with MO2 or MO6, depending on which one of them is not completed,
• then, all former procedures must be completed before procedures MO1 and MO3, belonging to phase 2, can be launched. When one of them, at least, is completed, MO5 can be launched. Then, all former tasks must be completed,
• MO7 can then be launched, as the only procedure in phase 5.

Example of sequencing on flow migration

As soon as the folder is revalidated, it can be accessed without use restriction (there can be some level of restriction if some postmigration operations have been postponed).

The only remaining task in to check and adapt some functional setups. This aspect is described in the documentation on the functional postrequisites.