Existing customers have been working on using GenevaERS metadata in related processes outside of GenevaERS. Our first contributor to open source, Sandy Peresie, took on the challenge of building a process which reads Geneva metadata XML file and produces an output of selected elements in COBOL. This was an experimental project, the first of the GenevaERS open source projects.
She chose to use Python 3.8.
To execute the process:
place the downloaded ascii/crlf VDP in the Data directory.
change the file name in the safrmain_poc.py
execute the file
Design specifications for this work in summary were:
View Column Records can be used to produce the expected output from the view.
To produce an input structure from the referenced LRs, do the following:
Use a LogicaRecord from the tree. Read its LOGRECID field. You can also get the LR name from this section.
Then in the LRField records look for fields that have that LOGRECID (that is treat it as a foreign key… a back reference) you will find there the LRFIELDID and name of the field and its start position within the record. (also an ordinal number) There is a complication here if the field redefines another at the same space. Look at the redefines. Suggest that you ignore redefined fields as a first iteration.
Now that you have the LRFIELDID you can use the LR-Field-Attribute section of the XML In there you will find the data type of the field FLDFMTCD, its length, sign, number of decimal places etc.
The following is a screen shot of the output she produced from her initial prototyping efforts focused on step 1.
Thanks for your work Sandy! You’ve started the ball rolling on GenevaERS Open Source Contributions!
Welcome to the training course on IBM Scalable Architecture for Financial Reporting, or SAFR. This is module 9, “Metadata Creation.”
Slide 2
This module provides you with an overview of logical records and files, physical files, and methods for creating and then associating logical records, logical files, and physical files. Upon completion of this module, you should be able to:
Create physical file definitions
Create logical file definitions
Associate physical files with logical files
Create logical record definitions, and
Associate logical files with logical records
Slide 3
This module describes creation of logical records, logical files, and physical files. Simple SAFR processes typically include the use of the following:
A logical record to describe the input record
A logical file to describe a group of input files
A physical file to describe an input file
Specifying a logical file and a physical file for output is optional.
Slide 4
Recall from an earlier lesson that:
A physical file definition describes a data source. Examples include customer files and order files.
A logical file definition describes a collection of one or more physical files.
A logical record definition, or LR, describes a record layout. In COBOL programs, record layouts are often found in copybooks. In relational databases, they are found in table definitions.
Slide 5
To create a physical file, you begin by opening the Administration menu, selecting New, and then selecting Physical File. To select and edit an existing physical file, click the Physical File icon in the navigation pane, and then double-click the appropriate physical file to select it from the list.
Slide 6
Begin by giving the physical file a descriptive name. All SAFR components must be uniquely named within an environment. The names may be up to 48 characters long, starting with a letter, and composed of letters, numbers, and the underscore (_). It is a good practice to add an underscore and the letters “PF” at the end of the name. This is not required, but it helps differentiate physical files from similarly named logical files and logical records. Typical names would be “CUSTOMER_PF,” “ORDER001_PF,” and “US_SALES_PF.”
Slide 7
Next, open the File Type list. Most SAFR processes use disk files. Select Tape File only when SAFR high-speed tape processing is required. Pipes and tokens are special internal SAFR virtual files that are never written to disk or tape. Find out more about them in the “Physical File” subsection of the “Metadata” section of the SAFR help documentation, which can be opened from the Help menu of the Workbench.
Slide 8
Next, open the Access Method list. Again, typical SAFR processes use sequential – standard access. Information about other types of access methods can be found in the SAFR help documentation.
Slide 9
The Read User-Exit Routine field, the parameters field, and the Comments field are optional. Read user-exits are custom-developed programs called by the Extract Engine to read data from files for input to SAFR, rather than the engine reading the files directly. Parameters may be passed to the exits when they are called.
Slide 10
If source data resides in an IBM® DB2® database, the “DB2 via SQL” access method or the “DB2 via VSAM” access method is chosen. This SQL section of the Physical File tab allows you to define the DB2 connection. Details about how to set up a DB2 connection can be found in the SAFR help.
Slide 11
The Dataset tab allows for specification of the input and output parameters for physical files. For input data sets, SAFR processes typically use the input DD name. This 8-character name should match what is placed in the JCL for the Extract phase step of the Performance Engine.
The data set name (or DSN), read disposition (or ReadDISP), and record length parameters are optional. If the actual fully qualified data set name is provided and the input DD name is not found in the JCL, the Extract Engine attempts to dynamically allocate this file. If no disposition is specified here, it defaults to Share.
Slide 12
The Output DD Name field enables you to specify the DD name that will receive output from a view. The other output data set parameters, which correspond to typical JCL file parameters, may be used for dynamic file allocation. Depending upon your operating system configuration, the Performance Engine may not be able to dynamically allocate output files.
Slide 13
When the physical file is saved, it is assigned a unique SAFR ID. The time stamps show when the file was created, and when it was last updated.
Physical files are contained within one or more logical files. A list of the associated logical files is presented on this screen for reference. The association of a physical file with a logical file can be done on the Logical File screen, which will be shown later.
Slide 14
Depending on how Workbench security is set up, the Workbench menu bar displays either the Administration menu (for system administrators) or the Components menu (for non-administrators). To create a logical file, you begin by opening the Administration or Components menu, selecting New, and then selecting Logical File. To select and edit an existing logical file, click the Logical File icon in the navigation pane and then double-click the appropriate logical file to select it from the list.
Slide 15
Remember that a physical file definition must have a unique name. A logical file definition must also have a unique name. The name may be up to 48 characters long, starting with a letter, and composed of letters, numbers, and the underscore (_). It is a good practice to add an underscore and the letters “LF” at the end of the name. This is not required, but it helps differentiate logical files from similarly named physical files and logical records. Typical names are “CUSTOMER_LF,” “ORDER_LF,” and “SALES_LF.”
You can use the Comments field to describe this logical file and its purpose.
Slide 16
Logical files are associated with one or more physical files. To make the association, click Add and select the physical file to be associated. After you click OK, the physical file is displayed in the Associated Physical Files area.
Slide 17
If we return now to the Physical File screen that we were discussing a few slides ago, we can see that the association is also shown here. In this case, CUSTOMER_LF is displayed as a logical file associated with the CUSTOMER_PF physical file.
Slide 18
After you save the logical file for the first time, an ID is assigned and the time stamps are updated.
The records in logical files can be described by many logical records. A list of the associated logic records is presented on this screen for reference. You can associate a logical file with an LR on the Logical Record screen. You must first define an LR before you make the association.
Slide 19
To create a logical record, open the Administration or Components) menu, New, and then select Logical Record. To edit an existing logical record, select Logical Record in thenavigation pane and then select the existing LR from the LR list.
Slide 20
Logical records must be given a name. As with PF names and LF names, an LR name may be up to 48 characters long, starting with a letter, and composed of letters, numbers, and the underscore (_).
Select Logical File for the type because this LR will be associated with a logical file. All LRs are inactive until they contain at least one field.
The lookup user-exit routine and parameters are used for custom-developed routines that are called instead of doing a lookup.
Slide 21
Click the LR Fields tab. The grid on this panel is used to create and maintain field definitions. Fields are defined with names, data types, start positions and lengths, and numbers of decimal points. All the data types discussed in the “I/O Processing and Data Types” module of this training course are available for each field.
To begin, select Add Field from the Edit menu. Then click the Add Field icon (the plus sign) on the Logical Record Editor toolbar or press Alt+Insert. This enables you to fill in values to complete the field definition. As with LR names, a field name may be up to 48 characters long, starting with a letter, and composed of letters, numbers, and the underscore (_).
Slide 22
When you save the logical record, the system automatically assigns a field ID to each field definition.
For a logical record to be the target of a join, you must define the structure of the primary key. The primary key can be composed of one or more fields. Click in the Primary Key cell for the first field in the key, and select 1. Select 2 for the second field in the primary key, and continue in this way until all key fields are defined.
The total length of all key fields in the LR is shown in the Total Key Length field.
Slide 23
Typically, as fields are added, the start position of the new field begins at the end position of the last added field. However, you can set field positions to overlap, using the Redefines check box. This check box allows modification of the Fixed Position field.
Use the number in the Total Length field to ensure that the length of the LR is correct when adding redefined fields.
In this example, the customer date of birth year and month field starts at the same position as the full customer date of birth field.
Slide 24
Scrolling right on the grid reveals additional field attributes that can be assigned. The Date/Time Format field designates how the date is stored in fields, if applicable. A check mark in the Signed check box indicates that the field contains valid sign data. If this field is used as a column in a view, the headings entered here (1,2, & 3) are used as the headings for the column in that view.
Slide 25
After you enter the field definitions, you click the Associated Logical Files tab. On this tab, you define which logical records contain records in the layout of the LR. Click Add and then select the logical files to be associated. After you click OK, the list is updated with the associated logical file.
Slide 26
This module presented you with an overview of logical records and files, physical files, and methods for creating and then associating logical records, logical files, and physical files. Now that you have completed this module, you should be able to:
Create physical file definitions
Create logical file definitions
Associate physical files with logical files
Create logical record definitions
Associate logical files with logical records
Slide 27
Additional information about SAFR is available at the web addresses shown here.
Slide 28
Slide 29
This concludes the Module 9. Thank you for your participation.
The following slides are used in this on-line video :
Slide 1
Welcome to the training course on IBM Scalable Architecture for Financial Reporting, or SAFR. This is module 3: “I/O Processing and Data Types.”
Slide 2
This module provides you with an overview of I/O processing and data types. Upon completion of this module, you should be able to:
Describe a logical record, and demonstrate its use
Describe the attributes of a view column definition
Name the various data formats that SAFR can read and write, and
Locate the input and output DD names in the metadata, and update the Extract phase JCL to include these names
Slide 3
To briefly review SAFR components, the Performance Engine reads source data using definitions in the SAFR Metadata Repository, and produces outputs. Logical records describe the inputs to the Performance Engine. The view definitions define the outputs, using the columns to specify the fields. Most outputs from SAFR are data files; however, SAFR is able to produce printable reports as well.
Slide 4
A logical record is similar to a COBOL copybook. It describes the input record structure, defining each field, the data it contains, and the format of that data. The Performance Engine reads input logical records, extracting data from fields to produce various outputs. It can transform the format of input fields depending upon the outputs required.
Slide 5
The value placed in the output field is defined by the column source properties. Output formats are defined by the column properties.
In this example, the constant with the value of 1234567890 will be written as a 10-byte zoned decimal field in position 11 of the file.
Zoned decimal format will be described later in this module.
Slide 6
The start position is calculated, and indicates where that field will be placed in the output file. It is dependent upon the length of prior columns. The data type, date/time format, decimal places, and visible and signed indicators are primarily important in views that produce files.
Slide 7
The highlighted attributes are primarily important for views that produce hardcopy reports. The column headings are actually printed on hard copy reports. In views that produce files, the headings are often used to document notes about the column. The other parameters will be detailed more in a later module.
Slide 8
The Performance Engine transforms data into numerous different types, many of which are not available as standard formats in COBOL or other languages. The choice of this attribute affects other column attributes, such as length, sign, decimal place, and mask, and thus is very important in building file output views. The following slides will help you understand how values are stored.
Slide 9
To understand these data types, we’ll use hexadecimal display or “hex” for short, to highlight differences between each data type. The example here is the zoned decimal formatted number 1234567890. In most mainframe editors, hex can be displayed by using the command hex or set hex while viewing file contents. The first line shows the display value. Below each character, two digits, one on top of the other, show the hexadecimal values. Note that a number can be stored in just one hexadecimal position.
Slide 10
The Signed check box determines whether a sign is stored with the number. In the example hex display for a zone decimal field type, an F means no sign is stored, a C means the number is positive, and a D means it is negative. The location of this sign, and in some cases, the actual value for positive and negative, is dependent on the data type.
All the data types discussed on the following slides can be defined as either input (in the logical record) or as output (in the view).
Slide 11
The tables here and on the following slides show the output value for different data types. Alphanumeric is a general format, but it cannot be used to maintain the sign of numbers. It is the same as a PIC X variable type in COBOL. It is the only format applicable for text data. The contents can be read in data files, but at the cost of wasted storage for numbers.
Slide 12
Zoned decimal is a general format for storing numbers with a sign indicator. It is the same as a PIC 9 or PIC S9 variable in COBOL. If the Signed checkbox is selected, a sign is stored in the top last hex position of the number, which makes the display version of the number unreadable. In this example, the sign is an opening brace. So, although the data is generally readable, it still requires more storage for numbers than other formats require.
Slide 13
The packed data type stores the number in a more compressed form and is commonly used in many languages, such as COBOL, where it is a COMP 3 variable. Because it is compressed, numbers can only be read in hex mode, each digit taking one hex position. The last hex position is reserved for the sign, regardless of whether the Signed checkbox is selected. Selecting the Signed checkbox determines whether “C” for positive or “D” for negative is placed in the position for the sign.
Slide 14
Binary is also a common compressed format, defined in COBOL as a COMP variable. Because binary format is compressed, the Signed checkbox has no impact on positive numbers.
Binary format is even more compressed than packed, and cannot be read without a conversion tool, such as a scientific calculator with hex mode.
Slide 15
SAFR provides two other masked formats, masked numeric and edited numeric. Both are similar to a masked, printable COBOL variable. The sign becomes a dash, which is not appropriate for subsequent processing on the mainframe.
After selecting the masked numeric data type in a column, you can can also select the specific mask to apply.
In the edited numeric format, a fixed mask is used, with no commas or leading spaces. This format is often used when preparing files for transfer to another platform for continued processing.
Slide 16
Users can define three other data types: binary coded decimal, sortable packed, and sortable binary. None of these data types have a COBOL equivalent.
Despite its name, binary coded decimal is not binary. It is more like packed, without a reserved place for the sign. It is often used to store dates and times.
Sortable packed and sortable binary can be sorted from large negative numbers, descending to zero, and continue to large positive numbers. Negative numbers in both these formats are difficult to decipher without a technical aid.
Slide 17
Masks override any value in the decimal field, regardless of whether the Signed checkbox is selected. In this example, the parentheses on the mask are used for negative numbers, and the output is signed even though the Signed checkbox is not selected. The mask has two decimal places, not the one decimal place defined in the Decimal Places field.
The Performance Engine aligns the decimal points from the input with the output and truncates or zero-pads if necessary. In this example, the input constant has three digits after the decimal place, but the mask has only two. The last digit is truncated in the output, and the negative constant is reflected in the parentheses.
Slide 18
Zoned decimal, packed, binary, and binary coded decimal all have implicit decimals. The output will have no explicit decimal point. However, again the Performance Engine aligns decimal points from input to output and fills with zeros if necessary.
In this example, the constant has three decimal points, but the output column specifies only one. Thus the final two digits, the 9 and 0, are truncated in the output.
Slide 19
In this example, we have replaced the constant input with a source field input. When the value of this field is moved from the input to the output, it will be transformed. It will be read as a 6-byte packed field and written as a 14-byte zone decimal. It will also have two decimal points added to it.
Note that you convert from one data type to another, the column widths may have to change. In this example, if the input 6-byte packed field were placed in a 6-byte zoned decimal output field, five digits would be truncated. On the other hand, if an input zoned decimal field is placed in a packed output field of the same size, space will be wasted.
Slide 20
This table shows the field limits for the different fields, and the number of decimals allowed in each field format. Each decimal place counts as a space in the maximum value of the number. For example, a packed field can be up to 31 digits in length, composed of 30 digits to the left and 1 digit to the right of the decimal point, or 22 digits to the left and 9 decimal places.
Note that, for the edited and masked numeric formats, the field length may be 256 bytes, but the practical limit is 31 digits for any data involved in a calculation, subtotal, data type transformation, or masking.
Slide 21
The Date/Time Format field further describes how dates and times are stored within the field. This enables certain date functions within SAFR.
In this example, we’ve added a date column to the prior view. The source contains the date in CCYYMMDD format, but the column output specifies a DD/MM/CCYY format. The Performance Engine moves the day from the last position to the first.
Note that, because the output requires slashes in the date, the column is a 10-byte field, not the 8 bytes on input.
Slide 22
DD names connect logical records and views to actual physical files read by the Performance Engine.
To find the DD name associated with an LR, we must first locate its associated logical file. The logical record for the view is shown in the View Source cell. Open the view in the Workbench and click in the blue View Source cell. The View Source Properties frame opens and displays the logical file, which in this case is ORDER_001.
Slide 23
To display logical and physical files, follow these steps:
Step 1: To see a list of all logical files, click the Logical File icon in the Navigator pane. The Logical File list opens.
Step 2: Scroll down the list until you find the ORDER_001 logical file, and then double-click the file name. The Logical File Editor opens.
Step 3: Find the physical file associated with this logical file by looking in the Associated Physical Files section of the page. The physical file here is named ORDER_001, which just happens to be the same as the name of the logical file.
Slide 24
To display the physical file and Input DD name, follow these steps:
Step 1: Click on the Physical File icon in the Navigator pane. A list of all physical files is displayed.
Step 2: Scroll down the list until you find the ORDER_001 physical file, and then double-click the file name. The physical file is displayed.
Step 3: To find the input DD name associated with this physical file, look in the Dataset section of the frame, and then the Input subsection. The input DD name in this case is ORDER001.
Slide 25
Now that you know that your view retrieves data from the DD name ORDER001, make sure that a DD statement for ORDER001 is included in your JCL and is referencing the appropriate data set.
Next let’s find the DD name for the output file.
Slide 26
View output is written to the physical file listed in the view properties. If the view does not require the Format phase, find the output DD name to place in the JCL by opening the view in the Workbench, navigating to the View Properties tab and clicking the Extract Phase tab.
Next, find the physical file in the Output File section of the tab. If the physical file value is blank, a new DD name will be generated at runtime. This name will consist of the letter “F” followed by the last seven digits of the view number, using leading zeros if required. If the physical file value is not blank, you must find the output DD name associated with the physical file. In this example, the output physical file is OUTPUT01.
Slide 27
Find the DD name for physical file OUTPUT01 by selecting Physical Files from the Navigator pane and then scrolling down in the Physical File list until you find the OUTPUT01 physical file. Double-click the file name.
The Physical File Editor frame for OUTPUT01 opens.
Find the output DD name associated with this physical file by looking in the Dataset section of the frame and then looking in the Output subsection. The output DD name in this case is OUTPUT01, which happens to be the same as the physical file name, but this isn’t always the case.
Slide 28
Now that you know that your view writes data to the DD name OUTPUT01, make sure that a DD statement for OUTPUT01 is included in your JCL and is referencing the appropriate data set.
Slide 29
This module provided an overview of I/O processing and data types. Now that you have completed this module, you should be able to:
Describe a logical record, and demonstrate its use
Describe the attributes of a view column definition
Name the various data formats that SAFR can read and write
Locate the input and output DD names in the metadata, and update the Extract phase JCL to include these names
Slide 30
Slide 31
This concludes the Module 3. Thank you for your participation.
