otp.DataSource#

class DataSource[source]#

Bases: onetick.py.core.source.Source

Construct a source providing data from a given db.

Parameters

  • db (str, list of str, otp.DB, default=None) – Name(s) of the database or the database object(s).

  • symbol (str, list of str, Source, query, eval query, default= onetick.py.adaptive) – Symbol(s) from which data should be taken.

  • tick_type (str, list of str, default= onetick.py.adaptive) – Tick type of the data. If not specified, all ticks from db will be taken. If ticks can’t be found or there are many databases specified in db then default is “TRD”.

  • start (datetime.datetime, otp.datetime, onetick.py.adaptive, default= onetick.py.adaptive) – Start of the interval from which the data should be taken. Default is onetick.py.adaptive, making the final query deduce the time limits from the rest of the graph.

  • end (datetime.datetime, otp.datetime, onetick.py.adaptive, default= onetick.py.adaptive) – End of the interval from which the data should be taken. Default is onetick.py.adaptive, making the final query deduce the time limits from the rest of the graph.

  • date (datetime.datetime, otp.datetime, optional) – Allows to specify a whole day instead of passing explicitly start and end parameters. If it is set along with the start and end parameters then last two are ignored.

  • schema_policy ('tolerant', 'tolerant_strict', 'fail', 'fail_strict', 'manual', 'manual_strict') –

    Schema deduction policy:

    • ’manual’ The resulting schema is a combination of desired_schema and database schema. Compatibility with database schema will not be checked.

    • ’manual_strict’ The resulting schema will be exactly desired_schema. Compatibility with database schema will not be checked.

    • ’tolerant’ The resulting schema is a combination of desired_schema and database schema. If the database schema can be deduced, it’s checked to be type-compatible with a desired_schema, and ValueError is raised if checks are failed. Also, with this policy database is scanned 5 days back to find the schema. It is useful when database is misconfigured or in case of holidays.

    • ’tolerant_strict’ The resulting schema will be desired_schema if it’s not empty. Otherwise, database schema is used. If the database schema can be deduced, it’s checked if it lacks fields from the desired_schema and it’s checked to be type-compatible with a desired_schema and ValueError is raised if checks are failed. Also, with this policy database is scanned 5 days back to find the schema. It is useful when database is misconfigured or in case of holidays.

    • ’fail’ The same as ‘tolerant’, but if the database schema can’t be deduced, raises an Exception.

    • ’fail_strict’ The same as ‘tolerant_strict’, but if the database schema can’t be deduced, raises an Exception.

  • guess_schema (bool) –

    Deprecated since version 1.3.16.

    Use schema_policy parameter instead.

  • identify_input_ts (bool) – If set to False, the fields SYMBOL_NAME and TICK_TYPE are not appended to the output ticks.

  • back_to_first_tick (int, offset, otp.expr, Operation) – Determines how far back to go looking for the latest tick before start time. If one is found, it is inserted into the output time series with the timestamp set to start time. Note: it will be rounded to int, so otp.Millis(999) will be 0 seconds.

  • keep_first_tick_timestamp (str) – If set, new field with this name will be added to source. This field contains original timestamp of the tick that was taken from before the start time of the query. For all other ticks value in this field will be equal to the value of Time field. This parameter is ignored if back_to_first_tick is not set.

  • presort (bool, default= onetick.py.adaptive) – Add the presort EP in case of bound symbols. Applicable only when symbols is not None. By default, it is set to True if symbols are set and to False otherwise.

  • concurrency (int) – Specifies number of CPU cores to utilize for the presort By default, the value from otp.config.default_concurrency is used.

  • batch_size (int) – Specifies the query batch size for the presort. By default, the value from otp.config.default_batch_size is used.

  • desired_schema (type[str]) – List of <column name> -> <column type> pairs that the source is expected to have. If the type is irrelevant, provide None as the type in question.

Examples

Symbol can be a collection

>>> data = otp.DataSource(db='SOME_DB', tick_type='TT', symbols=['S1', 'S2'])
>>> otp.run(data)
                     Time  X
0 2003-12-01 00:00:00.000  1
1 2003-12-01 00:00:00.000 -3
2 2003-12-01 00:00:00.001  2
3 2003-12-01 00:00:00.001 -2
4 2003-12-01 00:00:00.002  3
5 2003-12-01 00:00:00.002 -1

Source also can be passed as symbols, in such case magic named column SYMBOL_NAME will be transform to symbol and all other columns will be symbol parameters

>>> # OTdirective: snippet-name:fetch data.symbols as a source;
>>> symbols = otp.Ticks(SYMBOL_NAME=['S1', 'S2'])
>>> data = otp.DataSource(db='SOME_DB', symbols=symbols, tick_type='TT')
>>> otp.run(data)
                     Time  X
0 2003-12-01 00:00:00.000  1
1 2003-12-01 00:00:00.000 -3
2 2003-12-01 00:00:00.001  2
3 2003-12-01 00:00:00.001 -2
4 2003-12-01 00:00:00.002  3
5 2003-12-01 00:00:00.002 -1

Default schema policy is tolerant.

>>> data = otp.DataSource(db='NYSE_TAQ', tick_type='TRD', symbols='AAPL', PRICE=float, date=otp.dt(2022, 3, 1))
>>> data.schema
{'PRICE': <class 'float'>, 'SIZE': <class 'int'>}

>>> data = otp.DataSource(db='NYSE_TAQ', tick_type='TRD', symbols='AAPL', PRICE=int, date=otp.dt(2022, 3, 1))
Traceback (most recent call last):
  ...
ValueError: Database(-s) NYSE_TAQ::TRD schema field PRICE has type <class 'float'>,
but <class 'int'> was requested

Schema policy manual uses exactly desired_schema:

>>> data = otp.DataSource(db='NYSE_TAQ', tick_type='TRD', symbols='AAPL', PRICE=float,                                       date=otp.dt(2022, 3, 1), schema_policy='manual')
>>> data.schema
{'PRICE': <class 'float'>}

Schema policy ‘fail’ raise an exception if the schema cannot be deduced

>>> data = otp.DataSource(db='NYSE_TAQ', tick_type='TRD', symbols='AAPL', date=otp.dt(2021, 3, 1),                                      schema_policy='fail')
Traceback (most recent call last):
  ...
Exception: No ticks found in database(-s) NYSE_TAQ::TRD