Based upon the Active Project Status achieved in August 2021, the project team reviewed and updated it architectural direction. The following slides give a sense of progress made to date on the Spark Integration Efforts.
Main conclusions continue to be:
Code conversion of existing GenevaERS to Java for pre-processing increases product consistency
Spark adds significant capabilities to the tail-end of the GenevaERS Performance Engine, perhaps replacing the GVBMR88 Summarization Engine
Approaches for this integration require further development:
File based approaches may be more simple to code, but can Spark be configured to skip its Map phase?
Streaming integration may allow for reduced IO, but perhaps increase integration complexity.
The continued open Friday R&D Session (typically noon ET) will progress thinking on these points.
The following is a video extract of the Technical Steering Committee review of the slides which follow,
On August 12, 2021, the Technical Advisory Committee of the Linux Foundation’s Open Mainframe Project approved GenevaERS’s promotion from an incubation to active project. The following were the points made during the presentation, and a video extract of the meeting.
Initially developed as a product and consulting services asset from late 1980’s through early 2000’s when purchased by IBM as part of acquisition of PwC Consulting
Original name was GenevaERS until IBM Acquisition when it became known as Scalable Architecture for Financial Reporting or SAFR
Effectively a Map-Reduce Engine more than 10 years before development of Map-Reduce in 2004 by Google
Drives incredibly high throughput focused on z/OS on IBM Z Mainframes
A score of very large, enterprise customers over its history, continuing to be supported under contract for some customers
The mission of the GenevaERS Project, as stated in this document is:
1. Mission and Scope of the Project
a. The mission of the Project is to radically improve business systems using Business Event principles while processing significant data volumes to achieve :
● Accurate, fast and transparent outputs,
● Respond quickly to changing business needs, and
● Minimize costs over time.
b. The scope of the Project includes collaborative development under the Project License (as defined herein) supporting the mission, including documentation, testing, integration and the creation of other artifacts that aid the development, deployment, operation or adoption of the open source project.
(Technical Charter for GenevaERS a Series of LF Projects, LLC Adopted March 25, 2021)
And provided leadership for broader OMP initiatives, like the Open z/OS Enablement Project based upon our experience with shared environments. You can learn more about our relationship to our sister projects in this blog post.
We’re not done making progress though. In the coming weeks and months we are working to:
Release the Workbench, Run Control, and Performance Engine code bases to Github.
Begin to convert the GenevaERS Documentation to a new Github Home.
Continue to explore potential deeper integration with Apache Spark and GenevaERS, as the Map on the Mainframe component.
Apache Spark Integration: “Map” on the Mainframe Phase
The team continues nearly weekly R&D efforts to explore tighter integration with Apache Spark. GenevaERS has many similarities to the Map-Reduce constructs of Apache Spark, preceding it by a decade or more.
The GenevaERS Extract Engine GVBMR95, is a parallel processing, machine code generation function that can resolve many queries or functions in one pass through the underlying data. The GenevaERS Summarization and Aggregation Engine, GVBMR88, is much more like the reduce phase in Map-Reduce.
The team believes there may be distinct advantages to using the GenevaERS Extract Engine for the Map phase in Map-Reduce, coupled with Apache Spark for the Reduce phase, using its extended functionality and capabilities.
The team holds a weekly R&D Session on most Friday’s at 2:30 EST on webex if you are interested in joining.
Would you like to get more out of your valuable data on z/OS? Consider applying GenevaERS to the problem. Use the GenevaERS e-mail list to start a discussion of the potential use case.
A daily scrum call is held Monday through Thursday at 5:00 EST on webex (not held on TSC meeting days, 2nd and 4th Tuesdays.) Our On Boarding Document will allow you to get connected to all the GenevaERS Resources.
Additionally here are opportunities, many marked in the GenevaERS Repo as Good First Issue, to explore involvement in the GenevaERS Community:
The GenevaERS project is teaming up and providing support for two other Open Mainframe Initiatives; Polycephaly and the Open z/OS Enablement or OzE project.
Polycephaly is intended to be a key technology in expanding access to mainframes, marrying two different development life cycle methodologies, distributed and z/OS. Polycephaly requires minimal z/OS system programming, and provides flexible development paths and options, moving from linear to non-linear development. It removes the need for separate development paths for distributed and z/OS workloads. Developers can develop on any platform, store to Git and Jenkins to deploy.
GenevaERS’s Performance Engine, which resolves scores of queries or processes in a single pass through a database, today is executed via standard JCL. Polycephaly opens the possibility for an updated execution engine, allowing use of Git and Jenkins commands to perform all the functions typically done within JCL. This may open up use of the Performance Engine to resources not skilled in z/OS commands and JCL.
Work to progress this investigation would include attempting to convert the GenevaERS model Performance Engine JCL to Polycephaly commands. Doing this work will expose the developer to a number of new and old technologies, building bridges in interesting ways.
Learn more about Polycephaly through this introductory video on GenevaTV.
The Open z/OS Enablement or OzE project grew out of the experience of establishing a community working environment for GenevaERS. The team found there are few places upon which to do Open Source community work. And so the team proposed to the Open Mainframe Project an approach to help solve the problems.
The vision of is lower the bar for companies and individuals to make z/OS computing resources available more broadly. Lack of access to z environments is a major impediment to the growth and innovation on the platform. Type of uses targeted include:
Open Source Communities and new software development efforts
Mentoring and new user growth, consistent with and attractive to those who use other public cloud learning opportunities
Experimentation and innovation on the edge of environment stability like the Raspberry Pi Model Impediments to these types of environments include: – Critical knowledge and support in sysprogs for z systems – Cost and control of donated resources (MIPs, software, storage, etc.) – Security and access control
The project intends to create code, processes and techniques which reduce these impediments and enable broader use and development of the Z platform.
You can learn more about the Open z/OS Enablement Project by watching this episode of GenevaTV.
Organization of the project continues, but much progress has been made. Check out the Community Repository on GitHub, and its Governance and Technical Steering Committee Checklist to see what’s been happening.
But don’t stop there….
The first Technical Steering Committee (TSC) Meeting, open to all, is scheduled for Tuesday, August 11, 2020 at 8 PM US CDT, Wednesday August 12, 10 AM WAST/HK time.
Project Email List: To join the meeting or to be kept up to date on project announcements, join the GenevaERS Email List. You’ll receive an invitation as part of the calendar system. You must be on the e-mail list to join the meeting.
The POC will be composed of various runs of Spark on z/OS using GenevaERS components in some, and ultimately full native GenevaERS.
The configurations run include the following:
The initial execution to produce the required outputs will be native Spark, on an open platform.
Spark on z/OS, utilizing JZOS as the IO engine. JZOS is a set of Java Utilities to access z/OS facilities. They are C++ routines having Java wrappers, allowing them to be included easily in Spark. Execution of this process will all be on z/OS.
The first set of code planned for release for GenevaERS is a set of utilities that perform GenevaERS encapsulated functions, such as GenevaERS overlapped BSAM IO, and the unique GenevaERS join algorithm. As part of this POC, we’ll encapsulate these modules, written in z/OS Assembler, in Java wrappers, to allow them to be called by Spark.
If time permits, we’ll switch out the standard Spark join processes for the GenevaERS utility module.
The last test is native GenevaERS execution.
The results of this POC will start to contribute to this type of architectural choice framework, which will highlight where GenevaERS’s single pass optimization can enhance Apache Spark’s formidable capabilities.
Note that the POC scope will not test the efficiencies of the Map and Reduce functions directly, but will provide the basis upon which that work can be done.
Open source is all about people making contributions. Expertise is needed in all types off efforts to carry of the POC.
Data preparation. We are working to build a data set that can be used on either z/OS or open systems to provide a fair comparison for either platform, but with enough volume to test some performance. Work here includes scripting, data analysis, and cross platform capabilities and design.
Spark. We want some good Spark code, that uses the power of that system, and makes the POC give real world results. Expertise needed includes writing Spark functions, data design, and turning.
Java JNI. Java Native Interface is the means by which one calls other language routines under Java, and thus under Spark. Assistance can be used in helping to encapsulate the GenevaERS utility function, GVBUR20 to perform fast IO for our test.
GenevaERS. The configuration we create we hope to be able to extract as GenevaERS VDP XML, and provide it as a download for initial installation testing. A similar goal with the sample JCL that will be provided. GenevaERS expertise in these fields is needed.
Documentation, Repository Work, and on and on and on. At the end of drafting this blog entry, facing the distinct chance it will be released with typos and other problems, we recognize we could use help in many more areas.
The focus for our work is this Spark-POC repository. Clone, fork, edit, commit, create pull request, repeat.
On a daily basis there is an open scrum call on these topics held at this virtual meeting link at 4:00 PM US CDT. This call is open to anyone to join.
The following are some initial thoughts on the next version of GenevaERS as an Open Source project might go:
Currently the only way to specify GenevaERS processes (called a GenevaERS “view”) is through the Workbench, which is a structured environment allowing specifications of column formats, values to be used in populating those columns, including the use of logic called GenevaERS Logic Text.
GenevaERS developers have known for years that in some cases a simple language would be easier to use. The structured nature of the Workbench is useful for simple views, but becomes more difficult to work with for more complex views.
In response, we propose enhancements to the front-end of GenevaERS for the following:
Open up a new method of specifying logic besides the Workbench, initially Java.
This language would be limited to a subset of all the Java functions as supported by the the extract engines.
The current Workbench compiler would be modified to produce a GenevaERS logic table from the Java code submitted to it.
Develop plug-ins for major IDE’s (Eclipse, Intellij) that highlight use of functions GenevaERS does not support in the language.
GenevaERS Performance Engine Processes should be able to construct a VDP (View Definition Parameter) file from a mix of input sources.
Doing this would allow:
Storage of GenevaERS process logic in source code management systems, enabling all the benefits of change management
Opening up to other languages; often extracts from GenevaERS repositories are more SQL-like because of the data normalization that happens within the repository
Taking in logic specified for Spark and other execution engines which conform to GenevaERS syntax, providing higher performance throughput for those processes
Begin to open the possibility of constructing “pass” specifications, rather than simply defined in execution scripting.
Perhaps creation of in-line “exit” like functionality wherein the specified logic can be executed directly (outside the Logic Table construct).
The GenevaERS Performance Engine uses the Logic Table and VDP file to resolve the GenevaERS processes (GenevaERS “views”).
Proposed enhancements include:
Expanding today’s very efficient compiler to more functions, to support greater sets of things expressed in the languages. This would include things like greater control over looping mechanisms, temporary variables, greater calculation potential, and execution of in-line code and called functions within a view under the GenevaERS execution engine.
If there is interest and capacity, we may even move in the longer term towards an alternative Java execution engine. This would be a dynamic java engine, similar to the GenevaERS extract engine today; not a statically created for specific business functions as discussed below.
View to Stand-alone Java Program: It is possible to consider creating utilities which translate from GenevaERS meta data to another language, each view becoming a stand alone program, which could simply be maintained as custom processes. This would provide a migration path for very simple GenevaERS Processes to other tooling, when performance is not important.
Multi-View Execution Java Program: A set of GenevaERS views (a VDP) could be converted to a single Java program, which produces multiple outputs in a single pass of the input file, similar to what GenevaERS does today. In other words, it is possible to look at how GenevaERS performs the one-pass architecture, isolates differing business logic constructs from each other, perform joins, etc., and write new code to do these functions. This would also provide performance benefits from learning from the GenevaERS architecture.
Dynamic Java Program: Today’s GenevaERS compiler which produces a Logic Table could be converted to produce a Java (or other language) executable. This might add the benefit of making the processes dynamic, rather than static. This can have benefits to changing rules and functions in the GenevaERS workbench, and some sense of consistent performance for those functions, and the potential benefit of growing community of GenevaERS developers for new functions.
These ideas will be discussed at an upcoming GenevaERS community call to gauge interest and resources which might be applied.
Existing customers have been working on using GenevaERS metadata in related processes outside of GenevaERS. Our first contributor to open source, Sandy Peresie, took on the challenge of building a process which reads Geneva metadata XML file and produces an output of selected elements in COBOL. This was an experimental project, the first of the GenevaERS open source projects.
She chose to use Python 3.8.
To execute the process:
place the downloaded ascii/crlf VDP in the Data directory.
change the file name in the safrmain_poc.py
execute the file
Design specifications for this work in summary were:
View Column Records can be used to produce the expected output from the view.
To produce an input structure from the referenced LRs, do the following:
Use a LogicaRecord from the tree. Read its LOGRECID field. You can also get the LR name from this section.
Then in the LRField records look for fields that have that LOGRECID (that is treat it as a foreign key… a back reference) you will find there the LRFIELDID and name of the field and its start position within the record. (also an ordinal number) There is a complication here if the field redefines another at the same space. Look at the redefines. Suggest that you ignore redefined fields as a first iteration.
Now that you have the LRFIELDID you can use the LR-Field-Attribute section of the XML In there you will find the data type of the field FLDFMTCD, its length, sign, number of decimal places etc.
The following is a screen shot of the output she produced from her initial prototyping efforts focused on step 1.
Thanks for your work Sandy! You’ve started the ball rolling on GenevaERS Open Source Contributions!