otp.DataSource#

class DataSource(db=None, symbol=utils.adaptive, tick_type=utils.adaptive, start=utils.adaptive, end=utils.adaptive, date=None, schema_policy=None, guess_schema=None, identify_input_ts=False, back_to_first_tick=0, keep_first_tick_timestamp=None, **desired_schema)[source]#

Bases: onetick.py.core.source.Source

Construct a source providing data from a given db.

Parameters

  • db (str, list of str, otp.DB, default=None) – Name(s) of the database or the database object(s).

  • symbol (str, list of str, Source, query, eval query, default= onetick.py.adaptive) – Symbol(s) from which data should be taken.

  • tick_type (str, list of str, default= onetick.py.adaptive) – Tick type of the data. If not specified, all ticks from db will be taken. If ticks can’t be found or there are many databases specified in db then default is “TRD”.

  • start (datetime.datetime, otp.datetime, onetick.py.adaptive, default= onetick.py.adaptive) – Start of the interval from which the data should be taken. Default is onetick.py.adaptive, making the final query deduce the time limits from the rest of the graph.

  • end (datetime.datetime, otp.datetime, onetick.py.adaptive, default= onetick.py.adaptive) – End of the interval from which the data should be taken. Default is onetick.py.adaptive, making the final query deduce the time limits from the rest of the graph.

  • date (datetime.datetime, otp.datetime, optional) – Allows to specify a whole day instead of passing explicitly start and end parameters. If it is set along with the start and end parameters then last two are ignored.

  • schema_policy ('tolerant', 'fail', 'manual') –

    Schema deduction policy:

    • ’manual’ means the schema will be exactly desired_schema.

    • ’tolerant’ If the schema cannot be deduced, use desired_schema. If the schema can be deduced and it lacks fields from the desired_schema, or it has a field with a type incompatible with a desired_schema field, raise a ValueError. Otherwise, use desired_schema with deduced fields added. Also, with this policy database is scanned 5 days back to find the schema. It is useful when database is misconfigured or in case of holidays.

    • ’fail’ If the schema cannot be deduced, raise an Exception. If the schema can be deduced and is incompatible with desired_schema, raise a ValueError.

  • guess_schema (bool) –

    Deprecated since version 1.3.16.

    Use schema_policy parameter instead.

  • identify_input_ts (bool) – If set to False, the fields SYMBOL_NAME and TICK_TYPE are not appended to the output ticks.

  • back_to_first_tick (int, offset, otp.expr, Operation) – Determines how far back to go looking for the latest tick before start time. If one is found, it is inserted into the output time series with the timestamp set to start time. Note: it will be rounded to int, so otp.Millis(999) will be 0 seconds.

  • keep_first_tick_timestamp (str) – If set, new field with this name will be added to source. This field contains original timestamp of the tick that was taken from before the start time of the query. For all other ticks value in this field will be equal to the value of Time field. This parameter is ignored if back_to_first_tick is not set.

  • desired_schema (type[str]) – List of <column name> -> <column type> pairs that the source is expected to have. If the type is irrelevant, provide None as the type in question.

Examples

Symbol can be a collection

>>> data = otp.DataSource(db='SOME_DB', tick_type='TT', symbols=['S1', 'S2'])
>>> otp.run(data)
                     Time  X
0 2003-12-01 00:00:00.000  1
1 2003-12-01 00:00:00.000 -3
2 2003-12-01 00:00:00.001  2
3 2003-12-01 00:00:00.001 -2
4 2003-12-01 00:00:00.002  3
5 2003-12-01 00:00:00.002 -1

Source also can be passed as symbols, in such case magic named column SYMBOL_NAME will be transform to symbol and all other columns will be symbol parameters

>>> # OTdirective: snippet-name:fetch data.symbols as a source;
>>> symbols = otp.Ticks(SYMBOL_NAME=['S1', 'S2'])
>>> data = otp.DataSource(db='SOME_DB', symbols=symbols, tick_type='TT')
>>> otp.run(data)
                     Time  X
0 2003-12-01 00:00:00.000  1
1 2003-12-01 00:00:00.000 -3
2 2003-12-01 00:00:00.001  2
3 2003-12-01 00:00:00.001 -2
4 2003-12-01 00:00:00.002  3
5 2003-12-01 00:00:00.002 -1

Default schema policy is tolerant.

>>> data = otp.DataSource(db='NYSE_TAQ', tick_type='TRD', symbols='AAPL', PRICE=float, date=otp.dt(2022, 3, 1))
>>> data.schema
{'PRICE': <class 'float'>, 'SIZE': <class 'int'>}

>>> data = otp.DataSource(db='NYSE_TAQ', tick_type='TRD', symbols='AAPL', PRICE=int, date=otp.dt(2022, 3, 1))
Traceback (most recent call last):
  ...
ValueError: Database(-s) NYSE_TAQ::TRD schema field PRICE has type <class 'float'>,
but <class 'int'> was requested

Schema policy manual uses exactly desired_schema:

>>> data = otp.DataSource(db='NYSE_TAQ', tick_type='TRD', symbols='AAPL', PRICE=float,                                       date=otp.dt(2022, 3, 1), schema_policy='manual')
>>> data.schema
{'PRICE': <class 'float'>}

Schema policy ‘fail’ raise an exception if the schema cannot be deduced

>>> data = otp.DataSource(db='NYSE_TAQ', tick_type='TRD', symbols='AAPL', date=otp.dt(2021, 3, 1),                                      schema_policy='fail')
Traceback (most recent call last):
  ...
Exception: No ticks found in database(-s) NYSE_TAQ::TRD