I am delighted to announce the release of Glassbeam Studio – the industry’s first Data Transformation and Preparation tool for complex machine data.
According to Gartner, by 2018, data discovery and data management evolution will drive most organizations to augment centralized architectures with decentralized approaches. Most business users and analysts will have access to self-service tools to prepare data for analysis.
While data transformation and preparation is a complex undertaking even for structured data, doing the same for multi-structured data increases the complexity by an order or magnitude. Multi-structured data requires parsing complex logs, handling the volume, velocity and variety in the logs, handling variations in the log format across software versions of the device generating the logs and many more such complexities. And once all this preparation, is done the data has to be transformed into meaningful relational structures so that it can easily be retrieved for visualization, searching or any other form of consumption. The prepared data could be an input to a search index, could act as a source for analytics or be the data set for machine learning.
The capability of data transformation, preparation and App development was always part of Glassbeam through our hyper scale platform SCALAR and our parsing language Semiotic parsing language (SPL) which was offered as a professional services engagement. Starting Q1 2016, I am really happy to announce the launch of our beta version of Glassbeam Studio, which provides a very powerful and easy to use interface for preparation, transformation and App development of complex machine data.
This has been one of the most interesting projects that we have worked on at Glassbeam. Before I go into the details of GB Studio, I would like to point you to a previous BLOG, where I had explained about the strong foundation that we have built through our platform and language for structuring complex, multi-structured logs from connected devices. While the first step was to build the foundation through a scalable platform with a powerful, feature rich language for log analytics, the next step in the roadmap was to build a studio to apply the rich features of the platform through a simple user interface. We have a very rich roadmap for GB Studio and the below chart explains our vision.
The building blocks of GB Studio is broken into three parts
- Data Transformation and Preparation – This is the foundation layer and we have some really cool technology being developed in this layer. This is the layer that handles all the complexities of log format detection, generating corresponding SPL, providing transformation capabilities, testing data quality and more. This layer also provides an IDE for developing parsers using SPL. IDE provides a source code editor, syntax checker, auto suggestion and allows you to view O/P of your code to help debug.
- App Visualization and Builder – Data prepared will be fed into this layer, which helps visualize the data. Indexing for search, dashboard development, Rules Engine, Machine Learning platform are some of the Apps that would be integrated to consume prepared data
- Cloud Enablement and Deployment – In this layer, we would provide the required tools to size and deploy the complete solution on cloud.
I am really happy to announce our beta release of Glassbeam Studio. You can read more about this product HERE. The beta release has many compelling features
Data Transformation and Preparation
- It can detect all the well-known time series formats like syslog, comma, pipe, tab, space etc separated files, single line or multi-line events
- Provides all the standard features of an IDE like code editor, syntax checking, auto code completion and visualization of parsed content
- Allows for merging and splitting of auto detected formats and also allows extraction of additional content using simple drag and select UI
- Allows for transformation of parsed content. You can also pivot rows of a column into its own columns in the database
- Create sessions from log data. Session combines related sequence of logs into meaningful groups, so that group level analytics can be performed apart from event level analytics. This is very useful for medical device and manufacturing devices.
App Visualization and Builder
- Creates required structure in both C* for analytics and indexes in Solr to enable search on the parsed content
- Is integrated to our workbench, so that you can play around with the data using our drag and drop analytics tool.
and many more.
We have an exciting roadmap ahead for Glassbeam studio. Based on our initial customer and analyst reactions, we believe that this product will revolutionize the way multi-structured logs from complex devices and time series logs of any devices can be structured and modeled for analytics.
Stay tuned for these exciting updates…