Folder validation: technical annex

Introduction

The folder validation operation is an operation launched from folder management in the reference folder.
It can be launched in batch mode. Its aim is to perform the operations to bring the folder in a usable state, i.e. a state where connection and normal operations are possible on the corresponding endpoint.

The folder validation is a complex process that handles several cases:

• complete creation of the folder if it does not exist,
• structure changes of a folder that already exists, when:
• upgrading the reference folder,
• changing the activity codes configuration,
• setting up new languages,
• activating new modules,
• updating an history folder...

The creation of an history folder is done from the Creation of Purge folder function.

The process description has several steps, and these steps are described here.

Process steps

1 - Pre-requisite for history folders

If a history folder exists for a given folder, its revalidation is done in a second step, and can even be done much later (if for instance the environment where the history folder is installed remains as a stand-alone environment). But a history folder can only be migrated after migration of the folder it is attached to.

The history folder revalidation is only available for a migration that goes from version 150, version 6, version7, or upgrade 8 to upgrade 9 or later. Beware that this requires a minimum level of patches or maintenances depending on the version.

The following list gives the prerequisites:

2 - Preliminary controls

The folder validation first tests the parameter of the validation, with the following actions:

• If a history folder is validated, the reference folder is set to the associated exploitation folder.
• The reference folder is checked (it must be at the current version level).
• If the folder already exists, we are in a revalidation process, if not, we are in a validation process.
• The activity code values are controlled:
  - if an activity code is not set in the reference folder, it is deactivated,
  - in case of a revalidation, the list of activity codes that have been changed is updated,
  - the list of legislation activity codes updated is managed separately (some set-up updates are linked to legislations),
  - the evaluated activity code values (through dependency formulas) are computed,
  - the specific activity codes present are identified and checked.
• Additional functional controls are done (for instance, the value of some parameters).
• For history folder revalidation, check some pre-requisites (the view ADOVALHIS must be present in the reference folder: this is the case if the right preliminary patch has been installed on versions 150, 6, 7 and 8).
• The length of miscellaneous tables is checked (datatype ADI).
• The version of the engine is checked in order to ensure that optimized upgrade (migration plans) can be executed.

If some of the preliminary checks fail, the revalidation process will stop and the log file will give the reasons for this failure.

3 - Preliminary step of validation

If the folder is revalidated, the folder is first switched to mono-connection mode (no user can connect to the folder during the revalidation). This is of course only possible if no other user is already connected, otherwise the revalidation fails.

Flags defining if elements in dictionaries will have to be validated. By default, the revalidation is set to No except if we are in folder creation. In the next steps, the flag can be set to Yes by one of the upgrade procedures.

4 - Directory / Database creation

If the folder is created, a set of directories will be created in the application folder, and the scripts that create the database user with the corresponding tablespaces are run as well.

If the folder is revalidated, a check is done and the additional directories can be created (for instance when a language is added or if a new directory has been defined in the reference folder).

If the validated or revalidated folder is an history folder, the local dictionary elements (local table description) are created according to the data already stored, the dedicated menu files and text files used for the user interface are generated.

5 - Data transfer and purge

If the migration plans have been enabled:

• The list of temporary tables is set:
  - every module returns a list of tables to be purged,
  - some of them are temporary and can be purged without any problem (ALISTER, AWRKLOG, ATMPTRA for supervisor for instance, or PWRKSTT,PWRKORDERS for the purchase module),
  - some of them are purged but will be reconstructed later. This includes tables such as BALANCE, BALANA, GLCONSO for the accounting module.
• The list of movement tables is set as well.
• An entry point allows specific developments to add their own tables to the list.
• The temporary tables are purged.
• The movement tables are copied by direct SQL transfer to the dedicated U* migration tables.
• The movement tables are then purged.

6 - Table migration

The tables are now updated using the AMAJ routine located in the DOSMAJ scripts. The principles are the following:

• The indexes and the triggers are deleted.
• The columns to be added are added to the tables.
• The new tables added to the new version are created.
• The AMAJ routine in DOSMAJ* scripts are run, in the right order: for every release gap, a dedicated DOSMAJ* can run. For instance, to go from release X1 to Xn, the intermediate DOSMAJ that might exist to go from X1 to X2, from X2 to X3... from Xn-1 to Xn will run successively. This processes can, for instance, fill a new column with the content of an old column that might disappear at the next step, or fill a new table with the content of an old table that might disappear at the next step.

7 - Dictionary updates and structure changes

The table dictionary is now updated and the revalidation of all the modified tables will occur. This will drop the standard columns that are no longer present in the version and reindex the existing tables.

The movement tables transformation is also managed by this step. But if the migration plans have been used, the movement tables will be empty and the update will be performed later.

8 - Deliverables and kit management

If deliverables or kits have been added, they are copied from the reference folder (this step is not performed on a history folder).

9 - Dictionary update

When revalidating a normal folder, the dictionary content is updated for all the elements. This includes tables, views, actions, screens, objects, windows, data types, action parameters, functions, reports, activity codes, general parameters, miscellaneous tables, purge formulas, system transactions, inquiries, navigations, transactions, memo codes, type prototypes, global variables, constants, context variables, sub-programs, formula assistant contexts, URLs, documentations, data models, classes, representations, help keywords, entry points, business object related dictionaries.

The update takes into account the changes that have been made (especially new activity codes, new legislations).

The tables are now revalidated according to the new dictionary definition: columns that are no longer present are deleted, indexes are recomputed.

10 - Parameter updates and data copy

If we are updating a non history folder, depending on the changes in the setup, some data might be copied from the reference folder (for instance if a language code has been added, if new setups are delivered in a table and copied). The possible operations are the following:

• The new parameters created in the release are copied and initialized if necessary.
• The new miscellaneous tables are copied and initialized.
• The content of new tables that are delivered with automatic copy or conditional copy (if the copy has been set) is copied from the reference folder in the folder.
• The data related to kits that have been enabled is copied from the reference folder.

At this step, a link control procedure is executed on all the table except the movement tables if the migration plans are used.

11 - Dictionary generation

Still for non history folder on this step, the dictionary elements are validated:

• The views are generated.
• The local menu files are generated.
• The screens, objects, windows, inquiries are validated.
• The vocabulary elements are validated.

For history folder updates, only the tables and views are generated and validated (all the other dictionary elements are inherited from the main folder).

12 - Post upgrade data updates

If the we are in an upgrade process for non history folders, the MAJ routine in DOSMAJ* scripts are run, in the right order: for every release gap, a dedicated DOSMAJ* can run.
For instance, to go from release X1 to Xn, the intermediate DOSMAJ that might exist to go from X1 to X2, from X2 to X3... from Xn-1 to Xn will run successively.
This process can, for instance, perform additional updates. This step runs for both history folders and normal folders, but the scripts executed can of course be different.

13 - Classes and representation synchronization

If we are upgrading a non history folder, the classes and representation links are updated and the validation flag is set to trigger the validation later.

14 - Final dictionary operations

Still on a non history folder validation, the following operations are performed:

• The transactions are generated.
• If we create a mono-legislation folder, additional mass changes are done on some key linked to multi-legislation.
• If the folder has been created by the validation, default parameter values are filled according to the reference folder, default periods and fiscal year values are set.
• Menu files are generated from menu profiles.
• Triggers are generated on the tables.
• The update flags on the dictionary elements are reset.
• The BI parameter and code generation are done.
• The index files of compiled scripts are regenerated.
• Miscellaneous status flags are updated.
• The current version number is updated.
• Finally, the folder is switched back to multi-user mode.

15 - Additional step for migration if plans are selected

This step is relevant only if we are in an upgrade process and if the migration plans have been selected (otherwise, the folder validation is finished).

At this step, all the main parameters and dictionary tables are up to date, but all the movement tables are empty, and the corresponding data is located in the U* tables.

Running the migration plan steps will transfer the data by massive insertion into the final movement tables. This can be done with several steps in parallel, and is defined by the migration plan setup.

When finally this step is finished, the folder is again ready for normal operation. The U* table remain with the movement data image before the migration. They can be purged later (for instance after a given period of normal operation, to be able to analyze further data inconsistencies that were not detected).

 They need to be purged before the next upgrade will be done. This is done with the function TRTMIGDEL.

Additional comments regarding history folder upgrade

The following limitations apply to application table upgrade in history folders:

• Only the table having a standard history rule are migrated.
• The post-migration process (step 12 above) are not executed on history folder, except for balance resynchronization.

The consequence is that the following V6 fields will remain empty on the historized data:

• FIY and PER on BUD, GCOMMIT, SINVOICE (Sales/AR), PINVOICE (Purchase/AP) and PAYMENTH tables.
• UOM in GCOMMIT and BUD tables.

You should also remind that the accounting cumulated tables (BALANCE, BALANA, BLACMM, BLAQTY) are erased in V6. The migration procedure resynchronizes the balance table. If you want to use balance historized data, the GACCENTRY* and GCOMMIT* tables must also have been historized.

1)     Update before structure modification

If the folder validation is carried out while the folder to be updated is in an earlier version to the reference folder version, this phase will call the standardised update processes that must be applied (if they exist) before the structure of the data can be updated. The update processes are executed successively when moving from one version to another that are not immediately consecutive.

2) Creation of the directories

Created here are the directories that do not exist in the folder hierarchy (for example, those dependent on the language if a new language is used in the folder).

3) Update of the tables in the dictionary

The dictionary tables (i.e. those whose type is X3 Dictionary) are revalidated if their structure has changed.

4) Copy of the texts

All the texts found in the ATEXTRA table (texts in the dictionary), are copied here from the reference folder to the folder being revalidated. The precise process is as follows :

• firstly the standard texts are copied (where the number is less than 100,000). All the standard texts existing in the folder being revalidated but not present in the reference folder, are purged.
• if using a 3 tier architecture and where a third tier folder is being revalidated from the reference folder (that is not the supervisor folder) in the second tier, the process is started by synchronising the custom/specific texts (where the number is greater than 100,000) by creating in the reference folder the texts in the folder being revalidated if they do not exist. Thus, in this case, there is the situation where the specific/custom texts are identical in the reference folder and in the folder being revalidated.
• then the specific/custom texts are copied from the reference folder (if any exist) to the folder being revalidated.

It should be noted that this copy is made for all the standard languages managed in the folder being validated (and also for the software installation language, even if this language is not managed in the folder being validated). In the specific/custom languages no copy process is carried out.

This text copy phase (including specific/custom) signifies that at the end of the copy, there will be an absolute concordance between the text numbers in the reference folder and the revalidated folder, except for texts that did not exist in the reference folder.

It should be noted that the synchronisation phase in a 3 tier architecture, was not automatically assured in version 130. The consequences not being neutral for the management of specific/custom texts. They were as follows :

• the specific/custom texts that were found in the reference folder were recopied to the folder being validated.
• as a consequence, if different specific/custom texts existed in the two folders (i.e. specific/custom developments had been carried out both in the reference folder and in the folder being validated), certain texts would have been replaced in the revalidated folder by those in the reference texts, with the loss of the texts.
• to compensate for this problem, a text resynchronization utility was supplied.

5) Copy the local menus and messages

The local menus are recopied here and the messages from the reference folder to the live folder, except in the following cases :

• if the local menu already exists in the folder being validated and is modifiable by the user (it is therefore considered that the can have modified them).
• if the message chapter falls between 160 and 164 (these are the chapters reserved to the custom/specific carried out at the lowest level).
• if the message chapter falls between 165 and 169 and the reference folder is the supervisor folders (when this is not the case, they are copied : these are the chapter reserved for the "vertical"specific/custom that are transferred to the 3rd level in the case of three tier architecture).
• if the chapter is protected by a specific activity code.

This copy is made for all the standard languages managed in the folder being validated (and also for the software installation language, even if this language is not managed in the folder being revalidated). In the specific/custom languages no copy process is carried out.

In this phase, the local menu dictionary is also updated following the same rules.

6) Update of the table dictionary

This update will be carried out for all the database tables except those of the dictionary. In principal there is a comparison in the dictionary for all these tables, between the reference folder and the folder being validated.

• if the table exists in the reference folder, but not in the folder being validated, it is created.
• if the table exists in the folder being validated and is not protected by an activity code will be deleted from the dictionary if it no longer exists in the reference folder (for safety reasons, the table itself will not be deleted from the database).
• if the table exists in the reference folder and in the folder being validated, but it is custom/specific (i.e. identified by an activity code greater than or equal to X), if will not be updated except in the case of vertical customizations/specifics transferred in a 3 tier architecture.
• if the table exists in the two folders without being marked as specific/custom, the same update logic will be follwoed : the fields and indexes that do not exist in the table of the folder being revalidated will be created, the modified fields and indexes not protected by a specific/custom activity code will be modified, the fields and indexes protected by a specific/custom activity codes are not touched except if they are vertical specifics/customizations and the standard fields and indexes that no longer exist in the reference folder will be deleted.

Warning, in this phase, only the dictionary is updated (the database tables will be revalidated in a later phase).

In addition, it should be noted the number of records that will have been entered in the header of each table in the folder being validated is not updated if it is greater than the result of the sizing calculation or the value defined in the reference folder.

7) Copy of other dictionary elements

The other dictionary elements are updated in the following order :

The rule remains the same : any element that does not exist in the folder being validated is created, any standard element that no longer exists in the reference folder is deleted, any specific/custom element is respect, except in the case of 3 tier architecture with vertical activity codes. This logic extends in the same way, to the sub-elements (i.e. a field in a standard screen protected by a custom/specific activity code is created if it does not exist in the folder being validated ; if it already exists, no update takes place).

In addition, some information exists that is never updated by the transfer from the reference folder (except of course if the dictionary element did not exist before the validation in the folder to be validated). The includes the following information :

• the name of the specific/custom and vertical processes referenced in the screens, the inquiries and the actions.
• the access codes and the control tables associated with the fields in the screens. These facts are in affect considered as being parameterization likely to be modified by a user during use, with the help of the access code assignment and control table assignment functions.
• the help key words starting with X, Y or Z in the screens.
• the actions SPE, SPX and the code greater than or equal to X (and their parameters).
• the parameters in the objects left list (key, order, standard options, selection keys and expressions, number of characteristics for the search, presence or absence of the link explorer), as well as the associated reports and the tick box for access to the statistics. These elements are considered as parameterization likely to be updated by the object personalization function.
• the options for the objects starting with X, Y or Z.
• for the reports, the group fields, access codes, the printer characteristics, the Active flags, the segmentation parameter, the mandatory batch flag, that are considered as being elements likely to be changed by the user within usage procedures.

In addition, it should be noted that the dictionaries are updated, but that the screen validation and in the general fashion the generation of the code associated with the dictionaries will be made in a later phase.

8) Update of the values for the activity codes

This starts with the deletion of all the lines in the ACTIV table in the folder (including those starting with X, Y or Z). The the lines are recreated from the new folder parameters contained in the ADOSACT table in the supervisor folder (with the value Active or Inactiveaccording to the case and the associated sizing parameters).

9) Parameter update

This starts with the update of the parameters dictionary in the following fashion : any parameter not existing in the folder being validated is created, any standard parameter that no longer exist in the reference folder is deleted, any specific/custom parameter that already exists is not touched, except in the case of 3 tier architecture with an activity code vertical, any modified standard parameter is updated.

The parameter values defined at the level of the folder in the reference folder are copied, if they do not exist in the folder being validated.

It is important to note that, unlike version 130, it is now possible to create the specific/custom parameters (starting with X,Y or Z) protected by activity codes in each of the standard chapters (before, it was necessary to create them in the dedicated chapters starting with X, Y or Z).

10) Update of miscellaneous tables

This starts with the update of the miscellaneous table dictionary in the following fashion :

• any miscellaneous tables not defined in the folder currently being validated are created (both the parameterization and contents)
• any standard miscellaneous tables (i.e. not marked by an activity code starting with X, Y or Z) that no longer exist in the reference folder are deleted (parameterization and contents).
• any specific/custom tables that already exist are not touched
• any standard miscellaneous table whose structure is modified is updated conforming with the description below.

The update of the structure for each miscellaneous table is copied to the codes contained in this table in the following fashion.

• if a field has been added (Alpha 1 or 2, Numeric 1 or 2) to the miscellaneous table definition, the value contained in the corresponding codes is transferred, if they exist in the reference folder.
• if a field has been deleted, all the codes in the table are returned to zero.

It should be noted that tables 901 to 999, reserved for the supervisor, are always recopied (parameterization and contents) from the reference folder (they are never considered as specific/custom).

11) Update of the purging and archiving formulas

The update principal is as follows : any formula not defined in the folder being validated is created, any standard formula that no longer exists in the reference folder is deleted, any specific/custom formula that already exists is not touched, except in 3 tier architecture and vertical activity code, any modified standard formula is updated for all the definition elements (i.e. the date, time, frequency as well as the Active flag is not modified).

12) Update miscellaneous elements

This is the update of the dictionary type : any element not defined in the folder being validated is created, any standard element that no longer exists in the reference folder is deleted, any specific/custom element that already exists is not touched, except in 3 tier architecture and vertical activity code, any standard element modified is updated.

The corresponding elements are :

13) Update the application tables

The revalidation of all the application tables where the structure has been modified in pahse (6) is then carried out by using valfil.

14) Execution of any scripts

If SQL scripts exist stored in the PCD directory of the reference folder, in the form of ASCII folders, starting with majdos, with the extension sql, these scripts are executed (in creation, the script root is credos). This is used to foresee the creation of automated views for example, that did not exist in version 130. It is important to note that these scripts :

• can contain the comments, the lines starting with the # character being ignored.

• can contain the folder name, in the form of a variable named %nomap% : the substitution of this variable will be made before the execution of the script.

• are limited to 100 lines (comments and empty lines excluded) of 250 characters. But it is however possible to execute several scripts, the scripts being executed in alphabetical order of the file names.

• are executed by means of the Adonix engine (the connection is as the user corresponding to the folder name and with the privileges accorded by the roles user_ADONIX and admin_ADONIX).

15) Copy of the data

The data copy phase is used to transfer the data in the tables coming from the copy folder. This copy is made, in the context revalidation, that in the newly activated tables in the case where for example a module has been activated that was not previously. Of course, this copy can be automatic or conditional (i.e. dependent on the information entered in the Init tab in the folder management).

16) Generation of the code

This phase corresponds to the validation of the different elements in the dictionary. Only the elements modified in the previous phases will be revalidated (with the exception because of a version change for example can impose a regeneration code). The revalidated elements are in the following order :

These validations are, if necessary, carried out for all the languages in the folder (and also in the installation language if it is not one of the folder languages).

17) Control of the links

This phase carries out a referential integrity control on the data copied during the folder update, if this has taken place (phase 15). It verifies the totality of the records in these tables. If the value of the fields linked to the other tables does not correspond to the value of an existing key, the copied record is deleted if the link is mandatory ; if the link is not mandatory, the record is recreated with an empty key value.

18) Update after structure modification

If the folder validation is carried out while the folder to be updated is in an earlier version to the reference folder version, this phase will call the standardised update processes that must be applied (if they exist) after the structure of the data can be updated. This includes for example the fixing of the default values in the new fields in the standard tables. The update processes are executed successively when moving from one version to another that are not immediately consecutive.

It should be noted that this prevents the creation of a field in a given version, it gives it a value in this update phase, then deletes it in the earlier version and is capable of carrying out these two version changes in a single migration (because the update phase will not find the field in question). It should be noted that this is not the standard case, field deletions being rare.

19) Transaction generation

This phase will regenerate the code associated with the transactions that can be parameterized, once the database elements (and in particular in template screens) have been modified. This phase can be forced in certain version changes.

20) Update of the miscellaneous parameters

This phase completes miscellaneous parameters : default language, name of the folder, active modules, Web flag, current version, test folder, update of local menu for the analytical axes if their number changes... These are in general the parameters that cannot be modified by the user. Other updates can take place during this phase in the case of the folder creation (creation of at least one user, initialisation of the sequence number counters, creation of the first financial years and periods...).

21) Validation of the user menus and creation of the local menu files

The user menus are regenerated in all the languages used in the folder : that of the administrator is carried out from the functions table that could have changed, those of the users are simply purged of the functions that no longer exist. It should be noted that the users other than the administrator do not benefit, for security reasons, from the new functions created during update.

In the same way, a sequential file is created per language, containing the local menus. This file is loaded at the time of the connection to the client workstations (this generation is made just before the validation of the screens in the case where the folder is managed for the Wed).

22) Update of the version information

In this phase, the PARAM.ini files and version.inf that contain the folder parameters and the current version number are created in the folder directory on the application server.

23) End of update

In this phase, the table containing the application locks (APLLCK) is emptied, the flags used during the update are updated, the folder parameters are updated with the new version number, the mode returns to multi-user and finally the validation log file is displayed if not in batch mode.

Error messages

The error methods likely to occur are extremely varied and it is impossible to establish an exhaustive list. These errors are shown in the validation log file. Not all are serious (for example, not being able to revalidate a screen because of lack of space simply means it will be necessary to manually revalidate the screen). But in any case, it is necessary to seriously examine the validation log and all the errors found before restarting the live use.

In the case of a serious error, it is possible that the folder is in an incoherent state that prevents its usage and even a connection to it, the folder being locked. It is then possible to unlock the folder with the corresponding maintenance function, but it is necessary to be aware that this operation only momentarily unblocks the situation in order to allow a diagnostic connection. In fact, it should only be considered if the folder can be used normally again. The procedure to be followed is them to resolve the problem at the origin of the incident (for example this could be a lack of disk space) and to relaunch the validation, which will normally restart where it was stopped. In the worst case situation, it will be necessary to reload the backup, which should have been created before starting the validation.

Updated tables

All the tables in the folder can be updated by this complex process.