Options Announces Partnership With Code Willing

Options Technology, the leading provider of global trading infrastructure to Capital Markets firms, has announced a new partnership with Code Willing, a leading financial data management service in the fintech industry. This partnership will bring to market a new quantitative trading service intended to help clients better control their data analysis costs.

From the press release:

“Code Willing’s technology – ranging from file systems with granular file-based permissioning to high-performance raw data capture services – plays a key role in a number of unique new solutions for the financial services industry that Options will be bringing to market over the coming months, as we continue to build out our global financial ecosystem.”

About Options:

Options Technology is the No. 1 provider of IT infrastructure to global Capital Markets firms, supporting their operations and ecosystems. Founded in 1993, the firm began life as a hedge fund technology services provider. Today, the company provides high-performance managed trading infrastructure and cloud-enabled managed services to over 200 firms globally. For more information, please visit www.options-it.com.

About Code Willing:

Code Willing is a leader in data management solutions for the financial industry. Built on 20+ years of experience in fintech and trading, Code Willing offers data management services, cloud analysis tools, low latency market data feed handlers and scalable high-performance file storage. For more information, please visit www.codewilling.com.

Code Willing’s Client Featured in Online Article about Reducing Compute Costs by 90%

Risk.net, a website dedicated to risk management in complex markets, recently published an article highlighting Quantbot Technologies’ innovative strides in Big Data when utilizing Code Willing’s CASFS+ product.

Quantbot presently uses this product to effectively and efficiently run a version of its quantitative investing platform on the cloud. By implementing CASFS+ into their strategy, they discovered a 90% reduction in cost to run their jobs. The article discusses the algorithm used in order to reduce their spending, along with some additional, effective tools available in CASFS+

And it’s not “Quants only.”. Having built something that can provide multiple businesses with a faster and more cost-efficient way to run jobs, Code Willing is currently in the works to offer CASFS+ to other businesses in the AWS Marketplace. You can even contact them here to set up a demo and learn more about the product.

Link to the news article here:

https://www.risk.net/investing/7856556/this-quant-firm-cut-90-of-its-compute-costs-heres-how

Increasing Upload Speed With CASFS

100x Faster Database Inserts by Code Willing

Code Willing has determined the most efficient way to upload large amounts of data into a database hosted on their custom-built file system.

Following extensive research, Code Willing greatly decreased the time it takes to insert massive amounts of data into a TimescaleDB SQL database hosted on their file system, CASFS.

During their experiment, Code Willing used two different types of uploading methods: upserting data using Pangres and copying the data by using Timescale-parallel-copy.

The data being used in this research was time-sequential financial data daily files, with each of them having 2,229,120 rows and 19 columns.

The control variables during the course of the investigation were the size of the financial data files remaining identical for both uploading tests and using Pandas to process the data before inserting the data into the database.

While using the Pangres method, it allowed for easy insertion from Pandas dataframes into PostgreSQL databases using their upsert method. The upsert method takes advantage of Postgres’ insert function and handles issues like duplicates, adding missing columns, and datatypes.

By using the Pangres method, the upload time for inserting the data was 14.23 minutes (853.8 seconds).

 

Upload Origin Cores Ram Workers Upload Method Parallel Processing Server_data relationship Time (minutes)
Python Notebook 8 64 1 Pangres No none 14.23

Following this result, Code Willing wanted to find a quicker and more efficient way to upload their data. That is where the second uploading method came in.

Timescale-parallel-copy is, “a command-line program for parallelizing PostgreSQL’s built-in `copy` functionality for bulk inserting data into TimescaleDB.”

Code Willing performed three different ways to insert their data using Timescale-parallel-copy.

The first test was where the server-data relationship was local. The result led to an upload time of 1 minute (60 seconds).

 

Upload Origin Cores Ram Workers Upload Method Parallel Processing Server_data relationship Time (minutes)
Command Line 8 64 1 Timescale-parallel-copy No local 1.00

 

For their second trial, they considered using parallel processing and increasing workers by 1 to decrease the upload time further.

This resulted in an upload time of .35 minutes (21 seconds). 

 

Upload Origin Cores Ram Workers Upload Method Parallel Processing Server_data relationship Time (minutes)
Command Line 8 64 2 Timescale-parallel-copy Yes local 0.35 

 

During the third trial, Code Willing set up a TimescaleDB on a separate server and ran it continuously. This is described in the variable “Server_data relationship” as “not local.”

With this change in place, everything else remained the same, including parallel processing, except for changing the number of workers. After testing 1, 2, 4, 6, and 8 workers, they determined 8 workers were the most efficient. The upload time with 8 workers showed .13 minutes (8 seconds).

 

Upload Origin Cores Ram Workers Upload Method Parallel Processing Server_data relationship Time (minutes)
Command Line 8 64 8 Timescale-parallel-copy Yes Not local 0.13 

 

Through this research, Code Willing improved Pangres’ upsert method 99.1%, going from 14.23 minutes to 0.13 minutes, by using TimescaleDB’s parallel copy functionality and implementing Ray’s parallel processing.

To break it down even further, this new method allowed them to copy 278,640 rows per second.

In the end, Code Willing’s experiment determined when uploading copious amounts of data, the most efficient and quickest way is by using the built-in “copy” function (like Timescale-parallel-copy), as well as parallel processing with large amounts of workers to decrease the upload time of time-sequential data into a Postgres database (TimescaleDB).