By Claus Murmann and Ben Merrett

Since releasing CME Group’s DataMine plugin on the Beacon App Store last year, many customers have taken advantage of our integration with dataset from the world’s leading derivatives marketplace. We have recently added three new Wizards to the plugin, making it easier to a) configure the plugin, b) set up a daily scheduled job to collect data on the desired ticker symbols, and c) backfill Beacon’s time series database with historical intraday price data for the desired symbols and date range. 

Using the configuration Wizard

The first step to add the CME DataMine plugin to your Beacon domain is to install the plugin from the Beacon App Store, available in the Market Data category. Just click on the app and select “Domain Install” to add this to your Beacon domain (you will need a user account with ‘infraadmin’ privileges to do this).

This will add the CME Datamine repository onto your Beacon domain. You will find the configuration Wizards inside the CME repo in your Beacon IDE workspace. Because the CME Datamine plugin is deployed as a git repository, you will be able to browse the code and see examples such as how to query data. This should help you get started quickly.

The first step to configure the CME Datamine plugin is to connect to CME Datamine and verify that you are an authorized CME Datamine user. The Configuration Wizard pops up a dialog box to collect your organization’s CME user credentials, which is then encrypted and securely stored in your Beacon Vault. The Wizard then logs in to CME Datamine and issues a basic data query to verify that the plugin is working fine. After this setup step, the plugin will simply read the credentials from Beacon Vault for all users and jobs running in your domain, providing you with a seamless developer experience.

Configuring data collection jobs

Next, use the Collector Configuration Wizard to list the ticker symbols for which you want to set up Bob jobs to fetch intraday data at regular intervals. The Wizard will generate the config file for a daily Beacon job to collect the data, but you will need to step out of the Wizard to check in and push the Bob job YAML file definition that the Wizard generates. 

After committing and pushing out your Bob job definition, you can click Recompile in the Wizard or from the Bob Dashboard to instantiate the newly defined Bob jobs. After this setup, the job will run daily to load the prior day’s historical price data for the symbols that you’ve configured.

For example if you selected ‘CL’, the job will load all available trades across each contract that traded the previous day e.g. CLZ2, CLF3, CLG3 etc. and store it into the following location:

/TimeSeries/<your domain>/RT/Energy/<ticker>

Backfilling with historical data

So now you are all set up to collect new data on an ongoing basis, but what about historical data? We’ve built a Wizard for that, too. Just like the Collector Configuration Wizard, the Backfill Configuration Wizard just needs the ticker symbols you want to backfill for, plus the desired data range you want to backfill for. When you click “Run”, you will see the rolling log as the Wizard kicks off a distributed set of queries, using Beacon’s elastic grid compute manager.

Caution: this is for intraday data, and so one week’s worth of historical data for a popular ticker like near month crude oil futures can be over half a million data points. 

Having backfilled the data, you can now use it with your Beacon apps, such as Beacon Plot and in the near future, Beacon Notebooks. There, you can quickly and easily build and test trading strategies against real datasets and get them from idea to production while opportunities are still ripe. 

Another use case is in the context of regular collection of end-of-day settlement data for automated risk analysis or P&L reports. Note that the settlement data jobs will be coming soon in a later release of the CME Datamine plugin, which implements Beacon’s new Market Data Framework for writing futures curves that can be used directly in running risk and P&L.

Time-series in Beacon Plot:

Example strategy backtesting a spread prediction model using Notebooks in Beacon IDE:

Making automation easier with Wizards

We used Beacon’s Wizard Framework to build these Wizards. You can use the Wizard Framework as well to automate common setup procedures and configurations. The Wizard Framework consists of library components, APIs, and designs that make creating modular, web-based Wizards in Beacon quick and easy.

For more information, watch the Wizards in action here.