Based upon the Active Project Status achieved in August 2021, the project team reviewed and updated it architectural direction. The following slides give a sense of progress made to date on the Spark Integration Efforts.
Main conclusions continue to be:
Code conversion of existing GenevaERS to Java for pre-processing increases product consistency
Spark adds significant capabilities to the tail-end of the GenevaERS Performance Engine, perhaps replacing the GVBMR88 Summarization Engine
Approaches for this integration require further development:
File based approaches may be more simple to code, but can Spark be configured to skip its Map phase?
Streaming integration may allow for reduced IO, but perhaps increase integration complexity.
The continued open Friday R&D Session (typically noon ET) will progress thinking on these points.
The following is a video extract of the Technical Steering Committee review of the slides which follow,
Extract of GenevaERS Technical Steering Committee Architecture Update Discussion
On August 12, 2021, the Technical Advisory Committee of the Linux Foundation’s Open Mainframe Project approved GenevaERS’s promotion from an incubation to active project. The following were the points made during the presentation, and a video extract of the meeting.
Extract of OMP TAC Presentation on GenevaERS Project Renewal
Presentation Points
Background
Initially developed as a product and consulting services asset from late 1980’s through early 2000’s when purchased by IBM as part of acquisition of PwC Consulting
Original name was GenevaERS until IBM Acquisition when it became known as Scalable Architecture for Financial Reporting or SAFR
Effectively a Map-Reduce Engine more than 10 years before development of Map-Reduce in 2004 by Google
Drives incredibly high throughput focused on z/OS on IBM Z Mainframes
A score of very large, enterprise customers over its history, continuing to be supported under contract for some customers
Website (GenevaERS.org, new Linux Foundation GenevaERS.Com soon)
Adopters: Continuing GenevaERS customers•Committers: 17 team members from multiple organizations
Next Steps
Public Build Processes
Enhanced Documentation
Renewed Spark Integration Efforts
Other Enhancements
Project Mission Statement
The Linux Foundation has formed an LLC to legally hold the GenevaERS Intellectual Property. The terms of the entity can be seen on this Open Mainframe Project GitHub site.
The mission of the GenevaERS Project, as stated in this document is:
1. Mission and Scope of the Project
a. The mission of the Project is to radically improve business systems using Business Event principles while processing significant data volumes to achieve :
● Accurate, fast and transparent outputs,
● Respond quickly to changing business needs, and
● Minimize costs over time.
b. The scope of the Project includes collaborative development under the Project License (as defined herein) supporting the mission, including documentation, testing, integration and the creation of other artifacts that aid the development, deployment, operation or adoption of the open source project.
(Technical Charter for GenevaERS a Series of LF Projects, LLC Adopted March 25, 2021)
And provided leadership for broader OMP initiatives, like the Open z/OS Enablement Project based upon our experience with shared environments. You can learn more about our relationship to our sister projects in this blog post.
Project Goals
We’re not done making progress though. In the coming weeks and months we are working to:
Release the Workbench, Run Control, and Performance Engine code bases to Github.
Begin to convert the GenevaERS Documentation to a new Github Home.
Continue to explore potential deeper integration with Apache Spark and GenevaERS, as the Map on the Mainframe component.
Apache Spark Integration: “Map” on the Mainframe Phase
The team continues nearly weekly R&D efforts to explore tighter integration with Apache Spark. GenevaERS has many similarities to the Map-Reduce constructs of Apache Spark, preceding it by a decade or more.
The GenevaERS Extract Engine GVBMR95, is a parallel processing, machine code generation function that can resolve many queries or functions in one pass through the underlying data. The GenevaERS Summarization and Aggregation Engine, GVBMR88, is much more like the reduce phase in Map-Reduce.
The team believes there may be distinct advantages to using the GenevaERS Extract Engine for the Map phase in Map-Reduce, coupled with Apache Spark for the Reduce phase, using its extended functionality and capabilities.
The team holds a weekly R&D Session on most Friday’s at 2:30 EST on webex if you are interested in joining.
Become Involved
Would you like to get more out of your valuable data on z/OS? Consider applying GenevaERS to the problem. Use the GenevaERS e-mail list to start a discussion of the potential use case.
A daily scrum call is held Monday through Thursday at 5:00 EST on webex (not held on TSC meeting days, 2nd and 4th Tuesdays.) Our On Boarding Document will allow you to get connected to all the GenevaERS Resources.
Additionally here are opportunities, many marked in the GenevaERS Repo as Good First Issue, to explore involvement in the GenevaERS Community:
The GenevaERS project is teaming up and providing support for two other Open Mainframe Initiatives; Polycephaly and the Open z/OS Enablement or OzE project.
Polycephaly
Polycephaly is intended to be a key technology in expanding access to mainframes, marrying two different development life cycle methodologies, distributed and z/OS. Polycephaly requires minimal z/OS system programming, and provides flexible development paths and options, moving from linear to non-linear development. It removes the need for separate development paths for distributed and z/OS workloads. Developers can develop on any platform, store to Git and Jenkins to deploy.
GenevaERS’s Performance Engine, which resolves scores of queries or processes in a single pass through a database, today is executed via standard JCL. Polycephaly opens the possibility for an updated execution engine, allowing use of Git and Jenkins commands to perform all the functions typically done within JCL. This may open up use of the Performance Engine to resources not skilled in z/OS commands and JCL.
Work to progress this investigation would include attempting to convert the GenevaERS model Performance Engine JCL to Polycephaly commands. Doing this work will expose the developer to a number of new and old technologies, building bridges in interesting ways.
Learn more about Polycephaly through this introductory video on GenevaTV.
Introduction to Polycephaly
OzE
The Open z/OS Enablement or OzE project grew out of the experience of establishing a community working environment for GenevaERS. The team found there are few places upon which to do Open Source community work. And so the team proposed to the Open Mainframe Project an approach to help solve the problems.
The vision of is lower the bar for companies and individuals to make z/OS computing resources available more broadly. Lack of access to z environments is a major impediment to the growth and innovation on the platform. Type of uses targeted include:
Open Source Communities and new software development efforts
Mentoring and new user growth, consistent with and attractive to those who use other public cloud learning opportunities
Experimentation and innovation on the edge of environment stability like the Raspberry Pi Model Impediments to these types of environments include: – Critical knowledge and support in sysprogs for z systems – Cost and control of donated resources (MIPs, software, storage, etc.) – Security and access control
The project intends to create code, processes and techniques which reduce these impediments and enable broader use and development of the Z platform.
You can learn more about the Open z/OS Enablement Project by watching this episode of GenevaTV.
And provided leadership for broader OMP initiatives, like the Open z/OS Enablement Project based upon our experience with shared environments. You can learn more about our relationship to our sister projects in this blog post.
Project Goals
We’re not done making progress though. In the coming weeks and months we are working to:
Release the Workbench, Run Control, and Performance Engine code bases to Github.
Begin to convert the GenevaERS Documentation to a new Github Home.
Continue to explore potential deeper integration with Apache Spark and GenevaERS, as the Map on the Mainframe component.
Apache Spark Integration: “Map” on the Mainframe Phase
The team continues nearly weekly R&D efforts to explore tighter integration with Apache Spark. GenevaERS has many similarities to the Map-Reduce constructs of Apache Spark, preceding it by a decade or more.
The GenevaERS Extract Engine GVBMR95, is a parallel processing, machine code generation function that can resolve many queries or functions in one pass through the underlying data. The GenevaERS Summarization and Aggregation Engine, GVBMR88, is much more like the reduce phase in Map-Reduce.
The team believes there may be distinct advantages to using the GenevaERS Extract Engine for the Map phase in Map-Reduce, coupled with Apache Spark for the Reduce phase, using its extended functionality and capabilities.
The team holds a weekly R&D Session on most Friday’s at 2:30 EST on webex if you are interested in joining.
Become Involved
Would you like to get more out of your valuable data on z/OS? Consider applying GenevaERS to the problem. Use the GenevaERS e-mail list to start a discussion of the potential use case.
A daily scrum call is held Monday through Thursday at 5:00 EST on webex (not held on TSC meeting days, 2nd and 4th Tuesdays.) Our On Boarding Document will allow you to get connected to all the GenevaERS Resources.
Additionally here are opportunities, many marked in the GenevaERS Repo as Good First Issue, to explore involvement in the GenevaERS Community:
The Technical Steering Committee meeting had previously discussed the proposed future architecture for GenevaERS. The video of this discussion is on the YouTube GenevaTV channel. Embedded below is an update on the GenevaERS architecture
