01. Overview
Moving data between applications is a complex, tedious and tricky operation. The MayFlower Data Mover automates many of the processes which make this work difficult. The following discussion leads a reader through a generic data integration exercise. Since most Data Mover Tasks are very similar in structure, the methodology can be used as a guideline for creating new Tasks. Refer to documentation for each individual task type and the in-form documentation (pop-up help under clickable field labels) of the tasks you are working with.
02. Getting Ready for the Project
Data Movement shouldn't be done without a significant amount of preparation. Here is a short list of some of the critical steps involved:
02.01. Understanding the Applications
The most common problems occurring in data movement are caused by an incomplete knowledge of the source and destination applications. The specialist developing a data movement scheme must know what all the fields in both applications are used for. He should know how they impact application integrity and user experience if the are not available or incorrect.
02.02. Decide What to Move
After a thorough knowledge of both applications has been attained, it is possible to start deciding which fields should be moved. The relationship between the fields should be examined closely.
02.03. Identify and Resolve Common Data Movement Problems
Here is a brief list of some of the more common data movement problems. The MayFlower Data Mover handles most of these easily. In the cases where the destination is a Notes database, Data Mover can perform a compute-with-form operation on each document after transfer to the destination. Though this operation can slow down your task, it can be invaluable when your target system has many fields which need to be computed with default values. Keep this in mind when you are building your strategy.
02.03.01. Missing Data - It is not uncommon for the destination application to require fields which can not be found in the source records. A strategy for populating these fields must be developed. You can create the fields on the fly using the Data Mover's support of the Notes Formula language. Sometimes a second Task (perhaps from a second data source) is required to completely populate all fields.
02.03.02. Differing Field Names - it is rare for two applications to use exactly the same field names. Example: The source data has the value of 'John Smith' in a field called 'FullName' and the destination application uses a field called 'UserName'. These fieldname translations can be handled through the Field Mapping database or the Notes Formula language.
02.03.03. Differing Data Types / Formats - often information stored as one data type in the source application must be converted to another data type for use in the destination application. Example: The source data contains a field called 'RecordNumber' which is of type Text and contains values such as '00013254'. The destination application stores record numbers as Integers. These type conversions can handled easily through the Notes Formula language.
02.04. Data Integrity Before Data Integration
It is recommended that the source data be checked for integrity before moving any data to a destination application. This is especially critical where the source is a Notes database. Notes applications often evolve over time and it is not uncommon for newer records to have fields which do not exist in older or historical records. Once the rules of a data movement strategy have been set, the source data should be checked to ensure that all critical fields exist and have appropriate values all of the same data type. This prevents countless headaches further along the process. Many data integrity issues can be solved without modifying the source data. Because changing source data can cause so many problems (unwanted replications, document history changes, etc...) it is often better to use the Data Mover's support of the Notes Formula language to modify the information as it is moved. If the destination database has critical key fields, they should also be checked for completeness. Major problems with data integrity may necessitate a complete reworking of your data movement strategy.
02.05. Divide Into Tasks
Use your knowledge of the Data Mover's capabilities to break a data movement strategy into manageable chunks. You may need to create a series of Tasks to be run in a certain order to get the job done.
02.06. Create a Test Environment
Just about the worst thing to happen that can happen to a production system is to botch moving data. It is highly recommended that a non-production copy of both source and destination data sets be made to use while developing a data movement strategy.
03. Applying the MayFlower Data Mover
The following is a generic list of steps which can be taken to build Tasks for the Data Mover to perform. Not all these steps will apply to all Task types. This is an overview only.
03.01. Locate and Connect
The machine running the MayFlower Data Mover will need to be able to get to all the data involved in your Data Movement strategy. The machine should have: Notes Client access to all Notes Databases, disk access to all ASCII and local XML files, access to all Internet-based XML data sources, ODBC access to everything else. The Notes ID being used with the Data Mover should have access to all necessary records in all the Notes Databases being used, ODBC table Owner names should be identified (where necessary) and ODBC connection names and passwords should be verified, etc...
03.02. Profile Everything
This step is not absolutely necessary, but it makes the creation of Tasks extremely easy so it is highly recommended. Use the 'Create Sentinel Profile' menu action to create profiles of all your Notes databases. Use Sentinel to create ODBC Source profiles for all your ODBC sources. Use Sentinel's Field Mapping feature to create Field Definitions for all your ODBC sources.
03.03. Create your Task Document
In the Administration database, compose an appropriate Task document whether it be an Integration Task, a Notes Tool Task or an Administration Task. In the header of the document, give the task a name and description. Note that tasks usually are handled in alphabetical order. We recommend using a numbering or lettering scheme to keep specific tasks and groups of tasks ordered so they are easy to find. Task names should be kept reasonably short. The description should contain a brief overview of what the task is doing, and what project it is being used for.
03.04. Select your Data Source
The left side of the first tab in the Task document is where you specify the source for the data you will be moving. For Notes and ODBC data sources, you should first select the database profile you will be using. This will make filling out the rest of the document much easier. As soon as you have selected a profile, helper buttons will appear with most of the fields to be filled out. Use these helper buttons to avoid typing and typo's whenever possible.
03.04. Select Records
After the data source has been selected, you will need to specify what records you would like to have Sentinel read for this Task. For ODBC data sources, you can use an SQL 'Select' statement. For Notes data sources, use a Notes Formula Language 'Select' statement. Leaving the selection criteria blank will usually cause all records to be read.
03.05. Select Fields
Once you have defined the record set to be read, you should select the fields you wish to move. For ASCII sources, you need to define each field explicitly. For ODBC and Notes fields, you may leave the field selections blank if you wish all fields to be moved.
03.05.01. If you are selecting specific fields, and you also wish to define new fields in your Notes Formulas (discussed below) then you MUST specify the names of these new fields in your field list.
03.05.02. If selected fields are specified, all fields in the source document will still be available for use in Notes Formula Before calculations. You do not have to list fields you do not wish to move.
03.06. Select your Data Destination
The top right of the first tab in the Task document is where you specify the destination for the data you will be moving. For Notes and ODBC data destinations, you should first select the database profile you will be using.
03.07. Specify the Document Matching Scheme
This is the most critical piece of the Task form. Here you tell the Sentinel how to determine whether or not to move a piece of data to the destination. You will need to specify an Import Type. If the Import Type is not 'Add All', additional steps are required to specify the key structure for matching documents between the source and destination databases. This is discussed in depth elsewhere in this documentation.
03.08. Build a New View
Key Matching can be tricky. When working with a Notes destination, it is often necessary (and easiest) to build a completely new view in the target database. Often this ends up being a flat view called 'import' and it is hidden from most users.
03.09. Set Further Options
The second tab in the Task document contains Import Options. These are often highly specific and should be filled out for each task you create.
03.10. Set Limits and Logging
Developing new tasks can often be made easier by limiting the amount of records to be analyzed or moved. If there are 200,000 records in your data source, you may only wish to work with 20 or less at a time as you develop and debug your task. Setting the logging level can be helpful as well. Sentinel's highest logging level will produce a record of each record which is moved and each field which is changed in the destination data. Though this is invaluable during task development, logging this much information consumes a lot of processing time and disk space. It is recommended that logging be minimized when a task is put into production use.
03.11 Write Notes Formulas
Notes Formulas can be applied to documents being moved at three different stages.
03.11.01. The 'Notes Formula Before' is run against an in-memory copy of the source record before Sentinel compares it to the destination database. This can be very useful for performing manipulations such as creating new fields, removing unneeded fields, combining fields, etc.... Most of your data movement problems can be solved here.
03.11.02. The 'Notes Formula After' is run against the in-memory copy of the source record after Sentinel has compared it to the destination database. This is used less frequently than the Formula Before. Most often this is used to do field roll-ups and field removals.
03.11.03. The 'Update Source Document' is run against the original source document. Great care must be taken when using this function since it will actually modify the contents of your source database. It is most often used for 'stamping' documents with date-time or other information relating to the Sentinel processes.
03.12 Task Debugging Cycle
Before moving on to the next step of your data movement strategy, each task should be thoroughly tested and debugged. Here are the most common steps.
03.12.01. Use the 'Run Now' button or the Sentinel Data Integrator to launch your task.
03.12.01.01. The 'Run Now' button is only usable when the database is open on the machine where the Data Mover is installed.
03.12.02. Look at the Run Log generated by Sentinel. It will either be displayed on the screen after the task has run or you can find it in the Administration database. Check to see that your task behaved as expected.
03.12.03. Examine the records which have been moved or updated by the Data Mover.
03.12.04. Adjust formulas, field selections, keys, import options, etc... as needed and retry.
03.13 Build Remaining Tasks
If other tasks are require to implement your data movement strategy they should be built and tested. The entire data-movement strategy should be thoroughly tested. Once you are confident that everything works, run a test against complete test copies of your source and destination data. Often this will turn up some issues you will need to fix before putting your strategy into production.
03.14. Create a Task Set Schedule Document
Task Set Schedule documents are easy to build.
03.14.01 Select the tasks to run and arrange them in the proper order.
03.14.02 Select the time period and frequence for this schedule to be run. Take into consideration the heavy-use periods of your destination database to prevent replication or save conflicts from occurring.
03.14.03 Set the 'Active Status' field to Disabled.
03.15. Manually run the Schedule
Use the 'Run Now' button or the Sentinel Data Integrator to launch your Task Set Schedule. Ensure that all tasks run properly.
03.16. Redirect to Production Data
Update all Task documents to use production copies of your source and destination databases.
03.17. Enable the Task Set Schedule
Set the 'Active Status' field to Enabled. This will allow the Data Mover to automate the execution of your tasks.
03.18. Ensure Sentinel is Running
The last step is to check that the Sentinel Data Integrator (sent32.exe) application is up and running full-time. Make sure that the scheduler is enabled. As long as the Sentinel scheduler is active, the Task Set Schedule will be run automatically. |