FIDAL - Financial Data Access Library

How-to add a data source driver

1.0 Introduction

This document explain how someone can contribute the support of a new data source driver. This document is intended for developers working on enhancing FIDAL, not for user of FIDAL. User should instead consult the List Of Data Source or the API documents.

FIDAL design attempts to make the data source driver to be loosely coupled. This allow to add/remove data source without large impact on the code shared by the whole library.

The drivers are found in the fd_source directory.

The data source driver is implemented by providing approximately 10 well defined functions. These are explained in fd_source.h and it is strongly suggested to read this file first.

2.0 Design

Responsibilities of the data source driver are:

	Provide an index of the category/symbol available. Done once at the data source driver initialization.
	Fetch the data for a given category/symbol at the requested periodicity or shorter (if asking for daily, can provide 30 min data, but not weekly data).

Responsibility of FIDAL (the caller of the data source driver) are:
Provide the "unified database" concept.
Build the FD_History structure.
Merge the index and data from multiple data source.
Do some data integrity check.
Transform the data to different periodicity.
When data for a given symbols is build using multiple data source, process slower fetching in multithread.

3.0 Multithread Consideration

FIDAL can be build with or without multi-thread support. This must be considered also in the data source driver.

Code to handle Multithread (like semaphore get/release) should not be in the single thread version. To make that possible, you can use the FD_SINGLE_THREAD definition as follow:

#if !defined( FD_SINGLE_THREAD )

/* ... add code here for handling multithread... */

#endif

The following functions are called from only one thread:

	initializeSourceDriver
	shutdownSourceDriver

All the other functions in the data source driver can be called by multiple thread. The initializeSourceDriver function is guaranteed to be called only once by only one thread. It might get called again only following a corresponding shutdownSourceDriver. The driver can assume that shutdownSourceDriver is called while no other thread will be in any of the other driver functions.

The following functions can be called by multiple threads:

	openSource
	getParameters

The usage of the "FD_DataSourceHandle *handle" is a very convenient way to isolate data between the thread. When the user does a FD_AddDataSource, the "openSource" gets called once. All subsequent call for this data source will refer to this "handle". For a given "FD_DataSourceHandle *handle" the following will be called as if from the same thread:

	getFirstCategoryHandle
	getNextCategoryHandle
	getFirstSymbolHandle
	getNextSymbolHandle
	closeSource

The following functions can be called by multiple-thread, even for the same "FD_DataSourceHandle *handle":

getHistoryData

The driver can assume that upon a "closeSource" call, no other thread will be in the other driver function for the same "FD_DataSourceHandle *handle".

4.0 Future Plan

Plan on long term for possible drivers capability are:

	provide also market data (last known bid/ask per ECN).
	provide historical tick data.

Plan on long term for the common features are:

	allow to easily build composite index
	alias of a category/symbol (like a unix soft link)
	re-categorize symbols base on correlation.
	provide last known bid/ask per ECN (using the same redundancy principle by using multiple data source).
	provide historical tick data.
	optionally transform tick data in price bar.
	Allow to replicate the complete unified database into a "local compact database" for speed efficient data read. This is intended for providing a read-only database for efficient back testing / optimization on a large amount of data. Will help also for duplication of DB in a distributed processing setup.