Sign-up and Commit to the Spark-GenevaERS POC

Organization of the project continues, but much progress has been made. Check out the Community Repository on GitHub, and its Governance and Technical Steering Committee Checklist to see what’s been happening.

But don’t stop there….

Get Connected

The first Technical Steering Committee (TSC) Meeting, open to all, is scheduled for Tuesday, August 11, 2020 at 8 PM US CDT, Wednesday August 12, 10 AM WAST/HK time.

Project Email List: To join the meeting or to be kept up to date on project announcements, join the GenevaERS Email List. You’ll receive an invitation as part of the calendar system. You must be on the e-mail list to join the meeting.

Slack Channel: Conversations are starting to happen on the Open Mainframe Project Slack workspace. Request access here, then when granted look for the GenevaERS Channel.

Spark POC Plans

The team continues to prepare the code for release. But that’s no impediment to our open source progress.

We’ve determined that the best way to help people understand the power of The Single Pass Optimization Engine is to contrast it–and begin to integrate it–with the better known Apache Spark.

Here’s what we plan to do work on over the next several weeks, targeting completion before the inaugural Open Mainframe Summit, September 16 – 17, 2020.

This POC was designed using these ideas for the next version of GenevaERS.

The POC will be composed of various runs of Spark on z/OS using GenevaERS components in some, and ultimately full native GenevaERS.

GenevaERS Spark POC Approach

The configurations run include the following:

  1. The initial execution to produce the required outputs will be native Spark, on an open platform.
  2. Spark on z/OS, utilizing JZOS as the IO engine. JZOS is a set of Java Utilities to access z/OS facilities. They are C++ routines having Java wrappers, allowing them to be included easily in Spark. Execution of this process will all be on z/OS.
  3. The first set of code planned for release for GenevaERS is a set of utilities that perform GenevaERS encapsulated functions, such as GenevaERS overlapped BSAM IO, and the unique GenevaERS join algorithm. As part of this POC, we’ll encapsulate these modules, written in z/OS Assembler, in Java wrappers, to allow them to be called by Spark.
  4. If time permits, we’ll switch out the standard Spark join processes for the GenevaERS utility module.
  5. The last test is native GenevaERS execution.

The results of this POC will start to contribute to this type of architectural choice framework, which will highlight where GenevaERS’s single pass optimization can enhance Apache Spark’s formidable capabilities.

Note that the POC scope will not test the efficiencies of the Map and Reduce functions directly, but will provide the basis upon which that work can be done.

Start Contributing

Open source is all about people making contributions. Expertise is needed in all types off efforts to carry of the POC.

  • Data preparation. We are working to build a data set that can be used on either z/OS or open systems to provide a fair comparison for either platform, but with enough volume to test some performance. Work here includes scripting, data analysis, and cross platform capabilities and design.
  • Spark. We want some good Spark code, that uses the power of that system, and makes the POC give real world results. Expertise needed includes writing Spark functions, data design, and turning.
  • Java JNI. Java Native Interface is the means by which one calls other language routines under Java, and thus under Spark. Assistance can be used in helping to encapsulate the GenevaERS utility function, GVBUR20 to perform fast IO for our test.
  • GenevaERS. The configuration we create we hope to be able to extract as GenevaERS VDP XML, and provide it as a download for initial installation testing. A similar goal with the sample JCL that will be provided. GenevaERS expertise in these fields is needed.
  • Documentation, Repository Work, and on and on and on. At the end of drafting this blog entry, facing the distinct chance it will be released with typos and other problems, we recognize we could use help in many more areas.

The focus for our work is this Spark-POC repository. Clone, fork, edit, commit, create pull request, repeat.

On a daily basis there is an open scrum call on these topics held at this virtual meeting link at 4:00 PM US CDT. This call is open to anyone to join.

Contributions from all are welcome.

Lucky 13: GenevaERS is an Official Open Mainframe Project project

On Thursday July 9th, GenevaERS was approved an Incubation Project under the Linux Foundation’s Open Mainframe Project.

Comments from the approving Technical Advisory Committee included (paraphrased):

  • “This project showcases the types of applications that Z is very important to.”
  • “Would like to see capabilities to cleanse data before feeding into Spark for machine learning.”
  • “Yes, intersecting with Apache Spark would be an interesting development.”
  • “This is exactly the kind of project we want to see”
  • “Most of the projects we are sponsoring are user interface related, supporting modernization. This has interesting transaction processing characteristics.”
  • “Looks like you already have a couple of people interested in participating/contributing…” The energy in the committee is “because this is a mature/sophisticated project.”

The following slides were used in the application discussion:

Additionally, the relationship of GenevaERS to Apache Spark was discussed, leveraging the content from the recent Proposed GenevaERS 5.0 Architectural Direction.

Application For Linux Foundation Project Status

The GenevaERS has applied for incubation status under the Linux Foundation’s Open Mainframe Project.

The application can be viewed here.

To graduate from Incubation Stage, or for a new project to join as an Active Stage project, a project must complete the proposal process plus:

  • Have committers from at least two organizations.
  • Have achieved and maintained a Core Infrastructure Initiative Best Practices Badge.
  • Explicitly define a project governance and committer process.
  • Have a public list of project adopters for at least the primary repo
  • Complete a code licensing scan to ensure licensing is inline with the OMP guidelines
  • Elect or appoint a project lead to represent the project on the TAC
  • Receive a supermajority vote from the TAC to move to accepted stage.

Proposed GenevaERS 5.0 Architectural Direction

The following are some initial thoughts on the next version of GenevaERS as an Open Source project might go:

Workbench

Currently the only way to specify GenevaERS processes (called a GenevaERS “view”) is through the Workbench, which is a structured environment allowing specifications of column formats, values to be used in populating those columns, including the use of logic called GenevaERS Logic Text.

GenevaERS developers have known for years that in some cases a simple language would be easier to use. The structured nature of the Workbench is useful for simple views, but becomes more difficult to work with for more complex views.

Specific Proposals

In response, we propose enhancements to the front-end of GenevaERS for the following:

  • Open up a new method of specifying logic besides the Workbench, initially Java.
  • This language would be limited to a subset of all the Java functions as supported by the the extract engines.
  • The current Workbench compiler would be modified to produce a GenevaERS logic table from the Java code submitted to it.
  • Develop plug-ins for major IDE’s (Eclipse, Intellij) that highlight use of functions GenevaERS does not support in the language.
  • GenevaERS Performance Engine Processes should be able to construct a VDP (View Definition Parameter) file from a mix of input sources.

Benefits

Doing this would allow:

  • Storage of GenevaERS process logic in source code management systems, enabling all the benefits of change management
  • Opening up to other languages; often extracts from GenevaERS repositories are more SQL-like because of the data normalization that happens within the repository
  • Taking in logic specified for Spark and other execution engines which conform to GenevaERS syntax, providing higher performance throughput for those processes
  • Begin to open the possibility of constructing “pass” specifications, rather than simply defined in execution scripting.
  • Perhaps creation of in-line “exit” like functionality wherein the specified logic can be executed directly (outside the Logic Table construct).

Performance Engine

The GenevaERS Performance Engine uses the Logic Table and VDP file to resolve the GenevaERS processes (GenevaERS “views”).

Specific Proposals

Proposed enhancements include:

  • Expanding today’s very efficient compiler to more functions, to support greater sets of things expressed in the languages.  This would include things like greater control over looping mechanisms, temporary variables, greater calculation potential, and execution of in-line code and called functions within a view under the GenevaERS execution engine.
  • If there is interest and capacity, we may even move in the longer term towards an alternative Java execution engine.  This would be a dynamic java engine, similar to the GenevaERS extract engine today; not a statically created for specific business functions as discussed below.

Alternative Engines

Existing customers have expressed interest in alternative execution engines for SAFR logic, as demonstrated in the Initial Open Source Work: Metadata Translation. The following are approaches that can be taken to these requests:

View to Stand-alone Java Program: It is possible to consider creating utilities which translate from GenevaERS meta data to another language, each view becoming a stand alone program, which could simply be maintained as custom processes. This would provide a migration path for very simple GenevaERS Processes to other tooling, when performance is not important.

Multi-View Execution Java Program: A set of GenevaERS views (a VDP) could be converted to a single Java program, which produces multiple outputs in a single pass of the input file, similar to what GenevaERS does today.  In other words, it is possible to look at how GenevaERS performs the one-pass architecture, isolates differing business logic constructs from each other, perform joins, etc., and write new code to do these functions.  This would also provide performance benefits from learning from the GenevaERS architecture.

Dynamic Java Program: Today’s GenevaERS compiler which produces a Logic Table could be converted to produce a Java (or other language) executable. This might add the benefit of making the processes dynamic, rather than static.  This can have benefits to changing rules and functions in the GenevaERS workbench, and some sense of consistent performance for those functions, and the potential benefit of growing community of GenevaERS developers for new functions.

Next Steps

These ideas will be discussed at an upcoming GenevaERS community call to gauge interest and resources which might be applied.