Changelog#

onetick.py python package#

[Unreleased]#

[1.53.0]#

Added#

  • otp.Source.modify_query_times

  • otp.Source.time_interval_shift

  • otp.Source.time_interval_change

[1.52.0]#

Added#

  • otp.config.show_stack_info

  • stack trace for all onetick.query EPs

[1.51.0]#

Added#

  • added otp.TestSession which sets up default values

[1.50.2]#

Fixed#

  • fixed SharesOutstanding timestamp out of bounds

[1.50.1]#

Added#

  • parameter max_expected_ticks_per_symbol for otp.run

  • otp.config.max_expected_ticks_per_symbol

[1.50.0]#

Added#

  • parameter password for otp.run

  • otp.config.default_auth_username, otp.config.default_password

[1.49.1]#

Fixed#

  • otp.CSV: support fixed-length strings and varstrings

[1.49.0]#

Added#

  • Support multiple endpoints in the servers.RemoteTS entity

  • Introduced LoadBalancing for RemoteTS

  • Introduced FaultTolerance for RemoteTS

[1.48.4]#

Changed#

  • rename start_time and end_time parameters to start and end in join_with_query and other places

[1.48.3]#

Fixed#

  • Sphinx documentation

  • Added ability import otp.corp_actions as an alias for otp.functions.corp_actions

  • policy parameter in the join_by_time takes lower case values

[1.48.2]#

Fixed#

  • fix inplace parameter for otp.Source.insert_tick()

[1.48.1]#

Fixed#

  • custom start and end time in evaluated first-stage queries

[1.48.0]#

Added#

  • Added support for Python 3.7-3.10, onetick.py.backports module added for all common backports.

[1.47.0]#

Added#

  • otp.agg.num_distinct

[1.46.1]#

Fixed#

  • remove null-characters when converting string to varstring

[1.46.0]#

Added#

  • otp.Source.insert_tick() method

[1.45.4]#

Changed#

  • do not create temporary directory in case otp.DB locations are specified by user

[1.45.3]#

Added#

  • sort() function for tick lists

[1.45.2]#

Added#

  • changelog page in docs

[1.45.1]#

Added#

  • otp.Source.dump: message <no data>

[1.45.0]#

Added#

  • ranking aggregation class and method

[1.44.0]#

Added#

  • exposed otp.OneTickLib

  • otp.OneTickLib().set_authentication_token() method

[1.43.2]#

Fixed#

  • make generic aggregation tests pass on newest onetick builds again

[1.43.1]#

Changed#

  • otp.DataSource: search days back to find schema with tolerant schema policy

Fixed#

  • last_not_empty_date() method will not go beyond locator boundaries

[1.43.0]#

Added#

  • more nsectime/msectime conversions in column’s .apply() method

[1.42.0]#

Added#

  • convert from/to otp.string in column’s .apply() method

[1.41.5]#

Fixed#

  • make generic aggregation tests pass on newest onetick builds

[1.41.4]#

Fixed#

  • Column.str.to_datetime() now preserve nanoseconds when accessor is the same column

[1.41.3]#

Fixed#

  • merging string and varstring column

  • concatenating varstring with string

[1.41.2]#

Changed#

  • join_by_time now checks types in on parameter columns only if check_schema is True

[1.41.1]#

Fixed#

  • node is not checked if result is empty

[1.41.0]#

Added#

  • Source.time_filter() method added to filter ticks by time

[1.40.1]#

Backward incompatible change#

  • join_by_time now raise TypeError if on parameter is set and sources have different types of this column

[1.40.0]#

Added#

  • added otp.agg.option_price aggregation

[1.39.0]#

  • support callback mode in otp.run

  • otp.CallbackBase class

[1.38.0]#

Added#

  • support remote:// in otp.query

[1.37.1]#

Fixed#

  • Fix otp.run() error when query property repository not initialized

[1.37.0]#

Added#

  • added OTP_DEFAULT_FAULT_TOLERANCE config value

[1.36.2]#

Fixed#

  • otp.agg.generic improvements

    • remove all_fields parameter from docs

    • save aggregation sub-query to the same file

[1.36.1]#

Fixed#

  • updated requirements.dev.txt with strict versions of sub-dependencies to shorten the installation time

[1.36.0]#

Added#

  • added CORRELATION aggregation

[1.35.3]#

Added#

  • otp.agg.stddev now supports biased parameter

[1.35.2]#

Fixed#

  • float to string conversion in float accessor methods

[1.35.1]#

Changed#

  • removed timezone warning when creating stubs database locator

[1.35.0]#

Added#

  • parameter default for otp.Operation.map()

[1.34.0]#

Changed#

  • ignore unsupported field types when inspecting database

Fixed#

  • detect otp.varstring type when reading database

[1.33.0]#

Added#

  • yield and copy_tick() for per-tick script

[1.32.0]#

Added#

  • otp.raw class for specifying raw OneTick expressions

Fixed#

  • converting string literals to OneTick syntax

Backward incompatible change#

  • hack with string surrounded by double quotes is removed

[1.31.0]#

Added#

  • otp.oqd.sources.SharesOutstanding source with is actually OQD_SOURCE_SHO EP

[1.30.0]#

Changed#

  • removed otp.utils.memoize

  • otp.utils module refactoring

[1.29.0]#

Added#

  • otp.string[...] and a shortcut for it otp.varstring which represent varstring

[1.28.0]#

Fixed#

  • otp.join_by_time now can be used in otp.agg.generic

Added#

  • parameter use_rename_ep for otp.join_by_time

[1.27.1]#

Fixed#

  • pandas.DataFrame in otp.Ticks() now supports nan on Windows

[1.27.0]#

Added#

  • add_prefix and add_suffix functions to otp.Source

[1.26.0]#

Changed#

  • use TmpDir instead of GeneratedDir to create database directories

Added#

  • parameter rel_path for TmpDir

Backward incompatible change#

  • other TmpDir parameters are changed to be keyword-only

[1.25.0]#

Added#

  • otp.agg.generic

[1.24.0]#

Fixed#

  • automatically set default db/symbol/times for queries where possible

[1.23.0]#

Fixed#

  • inplace parameter for Source.update()

Backward incompatible change#

  • default inplace logic changed to False

[1.22.0]#

Added#

  • added otp.Operation.__round__ method

  • fixed docs

[1.21.2]#

Fixed#

  • float('nan') behaves the same way as otp.nan

[1.21.1]#

Changed#

  • otp.functions.corp_actions() parameter adjustment_date now supports otp.datetime, otp.date, datetime.date, datetime.datetime, str types

[1.21.0]#

Added#

  • inplace parameter for Source.write()

  • support using otp.Columns as symbol and tick_type parameters

  • added all OneTick parameters to Source.write()

Fixed#

  • fix wrong merge logic for Source.write() when propagate_ticks=True

Changed#

  • unify DB.add(), DB.put(), Source.write(), otp.db.write_to_db() as much as we can

[1.20.0]#

Added#

  • added ability to use pandas.DataFrame in otp.Ticks()

[1.19.0]#

Fixed#

  • removed overriding of tempfile module’s private function

Added#

  • mkstemp, mkdtemp, ONE_TICK_TMP_DIR functions in otp.utils

Changed#

  • refactoring of tmp files and database creation logic

  • remove usage of SKIP_NAME_GENERATION and DISABLE_COOL_TMP_NAMES environment variables

[1.18.1]#

Fixed#

  • disable emulation in inner .apply() functions

[1.18.0]#

Backward incompatible change#

  • deleted ability to use built-in min/max functions for otp.sources

  • now only otp.math.min/otp.math.max should be used for otp.sources If there are any usage of built-in min/max functions for otp.sources, please, replace them with otp.math.min/otp.math.max.

[1.17.4]#

Fixed#

  • fixed path checking for PYTHONPATH

[1.17.3]#

Fixed#

  • calling inner functions with tick fields in .apply() method

[1.17.2]#

Fixed#

  • protected from accidentally adding new attributes to the otp.config object

[1.17.1]#

Added#

  • large_ints parameter for First and Last aggregations

Fixed#

  • support nsectime by First and Last aggregations

[1.17.0]#

Added#

  • OTP_DEFAULT_LICENSE_DIR and OTP_DEFAULT_LICENSE_FILE configuration environment variables to set the location of license files

[1.16.5]#

Fixed#

  • per-tick script local variables

[1.16.4]#

Fixed#

  • state variables can be updated with simple values

[1.16.3]#

Fixed#

  • nanoseconds precision is not lost when using min/max functions with datetime arguments

[1.16.2]#

Added#

  • otp.config.default_concurrency

  • otp.config.default_batch_size

Changed#

  • default concurrency is set to the number of cores on the machine, previously concurrency was disabled

[1.16.1]#

  • Jupyterlab-snippets support added with command jupyter onetick_snippets jupyterlab_snippets

[1.16.0]#

Changed#

  • change default config parameters

    • default timezone is changed from EST5EDT to local

    • default start time, end time, database and symbol are removed

  • otp.DEFAULT_* variables are fixed, but deprecated

[1.15.12]#

Added#

  • Columns.map() function that mimics pandas.Series.map()

[1.15.11]#

Fixed#

  • forbid using pseudo-fields in .table()

[1.15.10]#

Fixed#

  • support using operations with _TIMEZONE in otp.join

  • support using datediff operation in otp.join

  • bug with some columns filtered out with otp.join

[1.15.9]#

Fixed#

  • change default timezone for datetime functions

[1.15.8]#

Added#

  • otp.functions.save_sources_to_single_file()

[1.15.7]#

Fixed#

  • don’t use cache in otp.DB.tick_types()

[1.15.6]#

Added#

  • otp.meta_fields class

  • otp.expr class

  • otp.Operation.expr property

  • support for using otp.expr in otp.DataSource parameter back_to_first_tick

[1.15.5]#

Added#

  • otp.RefDB class

[1.15.4]#

Fixed#

  • support otp.string in tick sequences functions

[1.15.3]#

Fixed#

  • otp.DB.last_date now returns first no empty date

Added#

  • otp.DB.last_not_empty_date() function

[1.15.2]#

Changed#

  • delete legacy code with _CompareEmulator

[1.15.1]#

Changed#

  • numpy version from 1.19.5 to 1.23.0

[1.15.0]#

Added#

  • otp.date and otp.datetime timezone awareness

  • support otp.date, datetime.date and pandas.Timestamp in otp.Source.table()

  • support otp.date as absolute time in otp.Ticks and otp.Tick

  • support creating otp.datetime with otp.date

  • support adding otp.date as column

  • removed otp.types.datetime2str, added otp.types.datetime2expr instead

Fixed#

  • fix ott.time2nsectime() function

[1.14.48]#

Added#

  • support empty return in per-tick script

[1.14.47]#

Added#

  • otp.oqd.sources.OqdSourceDes and otp.oqd.funcs.corp_actions

[1.14.46]#

Added#

  • support many tick types in otp.DataSource

[1.14.45]#

Changed#

  • updated onetick-py-test==1.1.30

Fixed#

  • otp.config: added .get() method

  • otp.Source.write(): new default value for date parameter

[1.14.44]#

Fixed#

  • default end time is next day’s midnight

  • otp.date: add and sub methods

[1.14.43]#

Fixed#

  • otp.DB will raise exception when ticks do not fall into the time range determined by DAY_BOUNDARY_TZ

[1.14.42]#

Fixed#

  • sources.CSV now support relative CSV file path located inside CSV_FILE_PATH folder

[1.14.41]#

Added#

  • otp.state.tick_list, otp.state.tick_set, otp.state.tick_deque, otp.Source.execute

  • support iterating over tick sequences in per-tick script

[1.14.40]#

Added#

  • otp.remote decorator for apply functions used in ray remote context

[1.14.39]#

Added#

  • otp.oqd module with oqd eps and sources

[1.14.38]#

Added#

  • otp.by_symbol function that allows to split query by symbols and use result with unbound symbols

  • added ability to pass in the otp.CSV file buffer created using the otp.utils.file in remote execution environment

[1.14.37]#

Changed#

  • Improving otp.CSV types inspection, added timestamp field argument, mimics pandas arguments names=, converters= and dtype=.

[1.14.36]#

Fixed#

  • schema dates lookup interval increased to get 5 five days back

  • fixed operation with Nanoseconds and constant dt objects

[1.14.35]#

Fixed#

  • fix apply() method with brackets, change lambda token parser

[1.14.34]#

Fixed#

  • min/max aggregation object modification when used on datetime columns

[1.14.33]#

Changed#

  • Changed method for converting python code to OneTick’s per-tick script and case() function

  • Improving performance of .script() and .apply() methods in more complex functions

[1.14.32]#

Changed#

  • All defaults moved to otp.config.[…] . E.g. otp.config.tz or otp.config.default_start_time

  • Options can also be accessed like otp.config[‘tz’]

  • otp.DEFAULT_[…] are preserved for backward compatibility; otp.Source.DEFAULT_[…] no longer exist.

Added#

  • config.default_symbology to set default symbology for all databases created by onetick.py

[1.14.31]#

Added#

  • otp.DataSource ‘keep_first_tick_timestamp’ parameter

[1.14.30]#

Changed#

  • rearranged docs to exclude sphinx and jupyter dependencies from the dist package

Added#

  • script that helps to pack aritficats for OT integration

[1.14.29]#

Added#

  • otp.Source methods: ob_snapshot(), ob_snapshot_wide(), ob_snapshot_flat()

[1.14.28]#

Added#

  • time_series_type parameter for First, Last, FirstTime, LastTime, FirstTick, LastTick, HighTime, LowTime

Fixed#

  • inheritance parameters for HighTick, LowTick, HighTime, LowTime

[1.14.27]#

Changed#

  • onetick.docs moved into the onetick.py.docs

[1.14.26]#

Added#

  • otp.MultiOutputSource class to create queries that return multiple outputs

[1.14.25]#

Added#

  • Possibility pass timezone with Operation type to ‘str’ and ‘dt’ accessor for ‘to_datetime’ and ‘strftime’ method

[1.14.24]#

Added#

  • Sphinx doc: added overivew, getting started and concepts into the doc. Added pydocs to the data inspection api

  • otp.config[‘context’] config variable

Removed#

  • pytz dependency

  • old version of the data inspection mechanism

[1.14.23]#

Added#

  • otp.coalesce()

[1.14.22]#

Added#

  • fix adding string columns in otp.Source.script()

[1.14.21]#

Added#

  • otp.utils.TmpFile now use __del__() instead of weakref.finilize()

[1.14.20]#

Added#

  • Aggregations: otp.agg.ob_snapshot, otp.agg.ob_snapshot_wide, otp.agg.ob_snapshot_flat

  • Sources: otp.ObSnapshot, otp.ObSnapshotWide, otp.ObSnapshotFlat

[1.14.19]#

Added#

  • otp.SymbologyMapping source

  • symbology and show_original_symbols parameters for otp.Symbols

[1.14.18]#

Added#

  • Environment variable OTP_SKIP_OTQ_VALIDATION now force to skip __validate_onetick_query_integration()

[1.14.17]#

Fixed#

  • otp.DB: try to get schema from db again without using cache if we failed to get it the first time

[1.14.16]#

Added#

  • argument all_fields of otp.agg() could set policy to choose tick for all fields: first, last, high, low.

[1.14.15]#

Fixed#

  • otp.cut, otp.qcut: raise exception when number of labels is not equal to number of bins

  • handle case when using cut functions on one field several times

[1.14.14]#

Fixed#

  • added logic to support changed behaviour of newer OneTick builds with respect to timezones passed to otp.run()

[1.14.13]#

Fixed#

  • ‘keep_everything_generated’ property for the classes TmpFile, TmpDir and GeneratedDir to control tmp file cleanup

[1.14.12]#

Fixed#

  • ‘members’ default autodoc option

[1.14.11]#

Added#

  • ability to use otp.Operations in aggregations

[1.14.10]#

Added#

  • docs for datetime classes

Fixed#

  • otp.Year, otp.Quarter, and otp.Month datetime offsets

[1.14.9]#

Added#

  • added documentation

Fixed#

  • fixed wrong documentation substitution for aggregations methods

[1.14.8]#

Changed#

  • improved doctest for apply and __getitem__ on the Source

  • added ability to use external queries from the doctest_resources subfolder

[1.14.7]#

Added#

  • otp.math.pi() function

  • improved docs for all otp.math functions

[1.14.6]#

Added#

  • Source slice data[['x', 'y']:] now works like handy shortcut for Source.table(strict=False)

[1.14.5]#

Changed#

  • otp.Source.high_time and otp.Source.low_time deprecated (use otp.agg.high/low_time instead)

Fixed#

  • Field collision in Source.agg() with all_fields=True and non-empty group_by

[1.14.4]#

Added#

  • otp.DataSource (or otp.Custom) now have back_to_first_tick parameter.

[1.14.3]#

Fixed#

  • Proper error message for using full tick aggregation on column.

[1.14.2]#

Added#

  • new optional flag -i (–name-info) for jupyter onetick_snippets (shows snippets name tree)

[1.14.1]#

Fixed#

  • Fix updating datetime columns with datetime functions

[1.14.0]#

Fixed#

  • Inconsistency in UPDATE_FIELD EP changing column type with datetime functions

[1.13.0]#

Changed#

  • API documentation structure changed

[1.12.1]#

Fixed#

  • Warning instead of an error when using source with improper schema as an FSQ

  • Restored support for passing _SYMBOL_TIME query parameter to “join_with_query” method

[1.12.0]#

Fixed#

  • objects designed to be protected moved to protected space

[1.11.1]#

Added#

  • ability to use absolute time instead of offset in otp.Tick() and otp.Ticks()

  • ability to pass the whole source as a list of symbol parameters in join_with_query()

[1.11.0]#

Added#

  • Added otp.functions.cut() and otp.functions.qcut(), that mimics pandas.cut() and pandas.qcut()

[1.10.3]#

Fixed#

  • nodes history rebuilding in otp.Source.deepcopy() method

[1.10.2]#

Fixed#

  • Method tick_types in inspection DB class can accept None date then assign self.last_date if it is None, self.last_date also can be None and when we pass arguments to otp run we create ‘end’ via ‘date + timedelta(days=1)’ that triggered Exception. Now it passes ‘utils.adaptive’ start and end in the case when date is finally None.

[1.10.1]#

Added#

  • ‘_TIMEZONE’ meta column of the otp.Source object

Fixed#

  • If you use otp.run() with a query as graph file and symbols as otp.Source, timezone is now being passed to the symbols query correctly

[1.10.0]#

Added#

  • jupyter-onetick_snippets executable - configure snippets for nbextension snippets/snippets_menu

[1.9.0]#

Added#

  • Added scripts which allows running QueryDesigner.exe on Windows for test case and dashboard on Linux

[1.8.0]#

Added#

  • math module with OneTick math methods there

[1.7.1]#

Fixed#

  • Datetime fields are now correctly converted to integers when passed as filter conditions

[1.7.0]#

Added#

  • Support numpy==1.19.5 for python3.9

[1.6.1]#

Changed#

  • otp.agg. is now function and return aggregation instance

Added#

  • Median aggregation

  • Source.agg will now work if running=False and all_fields=True - first tick in bucket used for all_fields

  • method apply for all aggregations that allows to apply aggregation to Source

  • Source methods high, low, first, last, high_time, low_time, distinct know supports all parameters from relevant aggregation

[1.6.0]#

Changed#

  • all calls to _Source changed to self.class where possible

  • copy() function now create subclass instance instead of _Source

  • output_type_index parameter for merge(), join(), join_by_time() and apply_query()

  • all standard otp.Source subclasses now support node and **kwargs parameters for __init__()

  • from this point all subclasses of otp.Source must support node and **kwargs parameters for __init__()

Fixed#

  • fixed problem with otp.Source.drop_columns() deleting custom properties

[1.5.24]#

Fixed#

  • otp.eval(source_func, symbol) now passes the whole symbol (_SymbolParamSource()) object to the source_func and not only symbol name

  • Internal graphs for sources and evals used as merge symbols are now stored in the main query file and not as separate files

  • Source.distinct() now aligns schema properly to have only key fields after the aggregation

[1.5.23]#

Changed#

  • if keep_timestamp == False source aggregations won’t add any reordering

  • source aggregation add_sort parameter deprecated and has no effect anymore.

  • if keep_timestamp == True source aggregations won’t add redundant reordering and will drop ‘TICK_TIME’ field

[1.5.22]#

Fixed#

  • pass all parameters from otp.run to otq.run

[1.5.21]#

Fixed#

  • The .update method works properly with string constants as values, and starts to return a resulting object

[1.5.20]#

Fixed#

  • Fixed passing milliseconds with proper convertion into the constructor of otp.nsectime to support backward compatibility

[1.5.19]#

Added#

  • support setting nanosecond constant to a column

[1.5.18]#

Fixed#

  • allowed to be otp.Symbols without specified db, helpful when databases comes as symbols from symbols flow

  • Source.count() failed in case of no ticks

[1.5.17]#

Added#

  • otp.Source.deepcopy() method and otp.Source.copy(deep) parameter

Changed#

  • _NodesHistory class refactoring: saving history as classes, not as closures

[1.5.16]#

Fixed#

  • source columns after join_with_query(prefix=…)

[1.5.15]#

Added#

  • distinct to support list of keys

[1.5.14]#

Added#

  • day_of_week function for datetime accessor

[1.5.13]#

Added#

  • otp.agg.max and otp.agg.min time_series_type parameter

Fixed#

  • parameter iteration logic for _Aggregation class

  • several typos in other aggregation’s parameters

[1.5.12]#

Fixed#

  • otp.Custom won’t fail if wore than one db passed as symbol param

[1.5.11]#

Fixed#

  • otp.Symbols(…, keep_db=False) worked incorrectly when symbol name contained a colon

[1.5.10]#

Added#

  • OTP_BASE_FOLDER_FOR_GENERATED_RESOURCE env variable that control where to save generated resources

[1.5.9]#

Changed#

  • Removed marking of nan values as special

[1.5.8]#

Added#

  • 'where' parameter for join_with_query

[1.5.7]#

Added#

  • LocalCSVTicks function, to read ticks from local csv file and create ticks object from them

[1.5.6]#

Added#

  • Support dateparts in the Ticks offsets, ie Hour, Nano, Minute, but without expressions

[1.5.5]#

Added#

  • Ability to pass parameters to eval

  • otp.Source filtering supports eval

  • _DBNAME meta field

  • _TICK_TYPE meta field

[1.5.4]#

Fixed#

  • the .table does not extend schema with new columns if strict=False

[1.5.3]#

Added#

  • 'keep_fields_not_in_schema' option for the otp.functions.join() preserves fields from sources that were not in the schemas

[1.5.2]#

Added#

  • Support for CEP queries

  • Added the .write() method on the source

  • Added the .count() method on the source that returns number of ticks

  • Support the FLOAT type for return OneTick schema

[1.5.1]#

Added#

  • Ability to pass pd.DataFrame as a symbol list to otp.run() and Source.to_otq()

Changed#

  • run.sh script that is generated for test cases now uses explicitly passed port by default and not MAIN_TS_PORT

Fixed#

  • Fixed timezone conversions in database inspection

  • Fix setting day_boundary_tz on database creation

[1.5.0]#

Added#

  • Ability to pass expressions in the group_by parameter for aggregatations

  • Introduced the otp.DEFAUT_TZ, otp.DEFAULT_START_TIME, otp.DEFAULT_END_TIME, otp.DEFAULT_DB

  • They can be configured using the corresonding env variables: ‘OTP_DEFAULT_TZ’, ‘OTP_DEFAULT_START_TIME’, ‘OTP_DEFAULT_END_TIME’, ‘OTP_DEFAULT_TZ’

  • The otp.DEFAULT_TZ is set to EST5EDT by default, and OTP_DEFAULT_TZ controls every place that works with timezones

  • Exposed the otp.databases() function that returns all available databases, where each database has corresponding methods for getting available tick types, symbols, dates with data and schema

  • Reworked mechanic of schema deducing, currently it is based on the otp.databases(): it does not use symbols under the hood anymore, that radicaly speeds up the algorithm and makes it timezone agnostic

  • Added ability to pass callables as a query for the otp.run. It supports functions and methods that have only single parameter that reflects upcoming symbol

  • Added ability to pass otp.dt in the join with query

[1.4.20]#

Fixed#

  • Fix bool type for multiple ticks in otp.Ticks.

[1.4.19]#

Added#

  • Add ability to use flexible buckets for Source.agg method.

[1.4.18]#

Added#

  • keep_timestamp parameter for “first”, “last”, “high” and “low” functions of Source class.

[1.4.17]#

Added#

  • Added names for Source objects. Names are used for display purposes when saving queries to disk.

  • Ability to save all sub-queries (JWQ, evals etc) to the same otq file when running a query

  • get_query_parameter_list function of the query inspector

Changed#

  • Source objects are now always saved to disk as otq files when running.

  • to_graph function of the Source class is now deprecated for complex queries

[1.4.16]#

Added#

  • Add ''tw_average' aggregation

[1.4.15]#

Fixed#

  • Fixed ability to ommit the n_bytes parameter in the str.subst

[1.4.14]#

Added#

  • Add 'same_size' option to join sources with same size directly

[1.4.13]#

Fixed#

  • Fixed HighTime, LowTime aggregations can now accept column name

[1.4.12]#

Added#

  • Add 'all' option in join function

[1.4.11]#

Fixed#

  • Remove using the onetick.test.data module in tests

[1.4.10]#

Changed#

  • Remove uncessary pacakges from the requirements.txt

[1.4.9]#

Added#

  • Possibility to specify db_locations in db.DB with derived dbs

[1.4.8]#

Added#

  • Double check init OneTickLib in the otp.run

[1.4.7]#

Added#

  • Overloaded >> and >>= operators for sources that duplicate Source.sink() method

[1.4.6]#

Fixed#

  • Problem related in the get_schema that could return no results due the timezone

[1.4.5]#

Changed#

  • Ability to pass schema deduction policy to get_schema()

[1.4.4]#

Fixed#

  • Custom() now works correctly with empty actual schema

[1.4.3]#

Changed#

  • Changed hardcoded date to 2030

[1.4.2]#

Changed#

  • Set onetick-lib version to 1.0.4

[1.4.1]#

Added#

  • Ability to call custom onetick script

[1.4.0]#

Changed#

  • Rename Source.join method to unite_columns.

[1.3.118]#

Changed#

  • Removed warnings from otp.utils, they seems happend to quite often and annoying for end users

  • Moved onetick.test package to the 1.1.26

[1.3.117]#

Changed#

  • Migrate from pytz to dateutil in inner usages.

[1.3.116]#

Added#

  • Source.join method for joining several columns into the one string.

[1.3.115]#

Added#

  • date method to dt accessor, for date extraction from time fields

Fixed#

  • Fields and state variable now can be initialized with otp.date

[1.3.114]#

Fixed#

  • bucket_time argument to otp.Tick.

[1.3.113]#

Fixed#

  • min and max aggregations are now working at the same time on nsectime fields.

[1.3.112]#

Added#

  • start_time_expression and end_time_expression arguments to otp.run

  • start_time_expression, end_time_expression and query_param arguments to Source.to_df

Changed#

  • time2nsectime function moved to otp.types module

[1.3.111]#

Added#

  • DISABLE_COOL_TMP_NAMES env allowing to disable patching of tmp file name generator with coolname implementation

[1.3.110]#

Fixed#

  • nsectime columns loose nanoseconds after the min and max aggregations

[1.3.109]#

Added#

  • stddev aggregation

[1.3.108]#

Changed#

  • pandas requirement change to 1.1.4.

[1.3.107]#

Added#

  • __repr__ method to otp.datetime class.

[1.3.106]#

Fixed#

  • Problems with initialization and arithmetic operations written in science notation

[1.3.105]#

Added#

  • double comparision by cmp and eq methods

[1.3.104]#

Added#

  • otp.run method

  • Symbol parameters now accessible via Source.Symbol.parameter_name

  • otp.date object now can be created from otp.datetime, pandas.Timestamp or datetime.datetime

  • date method to otp.datetime object

  • to_str method to otp.date object

  • otp.datetime now can be created from another otp.datetime object

Fixed#

  • time-value column comparison

[1.3.103]#

Fixed#

  • Symbol argument of otp.Ticks now fills _SYMBOL_NAME value.

[1.3.102]#

Fixed#

  • comparision of time column with time constants is now possible

[1.3.101]#

Fixed#

  • the group_by method applied a pin for a single output that broke binding with nested queries

[1.3.100]#

Added#

  • method of class can be used as parameter of otp.eval

[1.3.99]#

Fixed#

  • dump method now add labels if columns parameter was specified

Added#

  • callback parameter to dump method to preprocess data before dumping

  • dump method now accept string as columns parameter in such case only one column with such name and label (if specified) will be printed.

[1.3.98]#

Fixed#

  • otp.eval used NestedOtq EP that lead to strange problems in case of using external query as a source for the otp.Custom symbols

[1.3.97]#

Added#

  • match_if_identical_times argument to join_by_time method

[1.3.96]#

Fixed#

  • str.upper and str.lower had int type instead of str

  • otp.Symbols had no option to filter by the tick type, was introduced the for_tick_type parameter; introduced usability, added pattern and show_tick_type as parameters

  • Fixed bug in the .dump method: a query with dump generated multiple outputs and .to_df() returned a random one

  • Added ability to fetch subschema from the schema property on the Source using the list of columns, ie data.schema[['QTY', 'PRICE']]

  • Fixed bug that query_inspector tried to parse a python file with otq.query_creator as a first stage query and generated QueryNotFoundError

  • Access to non existing column in the schema led to creating a new column in the schema with the float type

  • Deprecated TTicks

  • Added the strict parameter to the table equal to the keep_input_fields flag in OT

  • Added conversion from int to nsectime

[1.3.95]#

Added#

  • Added the ‘high_time’, ‘low_time’ aggregations (standalone box & compute)

[1.3.94]#

Added#

  • Introduced the ‘dump’ method

[1.3.93]#

Added#

  • Validation that path to Onetick binary is in $PATH environment variable on Windows

[1.3.92]#

Fixed#

  • Dateparts (otp.Second, otp.Milli and so on) are now correctly support initialization with difference between two lag operators

[1.3.91]#

Fixed#

  • Timestamp field assignment with _START_TIME and _END_TIME is now possible

[1.3.90]#

Added#

  • Support state variables assignment with otp.datetime

[1.3.89]#

Added#

  • Support a forward lag operator for add and update (via temporary column) fields

[1.3.88]#

Fixed#

  • derived dbs are now saved after tests with –keep-generated flag

[1.3.87]#

Added#

  • Support a column assignment with otp.datetime

[1.3.86]#

Added#

  • schema for the symbol param source

  • type conversion from nsectime to int

  • presort support for merge with bound symbols

Fixed#

  • ability to create a otp.dt from another otp.dt object

  • update the .Time column based on operation

[1.3.85]#

Added#

  • Support the TABLE EP as the .table method on the Source

Removed#

  • Custom database like TAQ_NBBO are removed from the package, because they are custom

[1.3.84]#

Added#

  • now it is possible to call methods (e.g. fillna, round, etc.) on expression not only columns

  • python way expressions for user filter, no need to compare with 0 or empty string\zero timestamp

Fixed#

  • string + operation now doesn’t fail with 3 or more operands

  • accessors now works on state variables

[1.3.83]#

Changed#

  • quotes in eval statements are now escaped with slashes, not expr as it was before

[1.3.82]#

Added#

  • adaptive tick type documentation

  • windows short path configuration description

[1.3.81]#

Changed#

  • add otp.eval function for specifying symbol and start/end time of evaluated query or Source

  • otp.query now supports expressions as parameter

Fixed#

  • now it is possible to create timezoned otp.datetime object from datetime with no default timezone

[1.3.80]#

Changed#

  • move onetick-py-test to 1.1.25 to grab cleaning unused file handlers

[1.3.79]#

Changed#

  • move onetick-py-test to 1.1.24 version

[1.3.78]#

Fixed#

  • passing otp.sources.query objects to the Custom as a symbol

[1.3.77]#

Added#

  • add pylama static code checker to CI

[1.3.76]#

Added#

  • add_passthrough flag to Source.to_graph() method

[1.3.75]#

Added#

  • added supporting per-tick-script logic through the .script method on the Source supports only if-else, adding column and return stateent

  • introduced BaseSchema as property of the _Source

  • introudced Source as alias for the _Source

[1.3.74]#

Added#

  • Ticks and Tick can handle special utils.adaptive value in tick_type argument to use a tick type from the sync node.

[1.3.73]#

Added#

  • apply_query and otp.sources.query now handle both queries with unbound and bound symbols (used to work only with bound)

[1.3.72]#

Added#

  • otp.datetime now has nanoseconds precision

  • otp.Nano and so on supports column and expression as the argument

[1.3.71]#

Added#

  • Added ability to add external users into the generated acl files

[1.3.70]#

Added#

  • process_by_group method to Source object

[1.3.69]#

Fixed#

  • Changed type of Exception in case if query would not found

[1.3.68]#

Fixed#

  • OMDSEQ field isn’t renamed anymore by join_by_time function

[1.3.67]#

Fixed#

  • Moved onetick-py-test version to get fix related to the using per-tick-script

[1.3.66]#

Added#

  • properties .db and .tick_type to the Custom source

[1.3.65]#

Added#

  • isin method for checking if column’s value in items

[1.3.64]#

Added#

  • ability to the join_by_time to set several leading sources

[1.3.63]#

Fixed#

  • otp.query param substitution is taken quotes and slashes into account

[1.3.62]#

Updated#

  • Speed up the Ticks() performance for simple cases

[1.3.61]#

Updated#

  • Moved conftest.py logic into the onetick-py-test==1.1.21, and moved dependency forward

[1.3.60]#

Fixed#

  • OTQ_PATH includes keep generated folder for sessions

  • tick_timestamp_type db property is set to NANOS by default

[1.3.59]#

API Change#

  • state variables are now collected in .state_vars field of the _Source

Remove#

  • otp.state.var method

Fixed#

  • stacking otp.funcs.join with rprefix specified doesn’t cause an error anymore

Fixed#

  • state variables weren’t been copied after copy, merge and join operation

[1.3.58]#

Added#

  • auto escaping in otp.query parameters

  • ability to use onetick.py expression as otp.query parameters

[1.3.57]#

Added#

  • otp.inf for double positive infinity

[1.3.56]#

Added#

  • Numeric and boolean constants as indexes of source will cause ValueError

[1.3.55]#

Added#

  • ability to pass tick-dependent expressions to join_with_query parameters start_time and end_time

[1.3.54]#

Added#

  • set_schema method to specify schema in python part

[1.3.53]#

Fixed#

  • query_inspector now works correctly with commented nodes and bound securities

[1.3.52]#

Added#

  • otp.Query can now work on queries without unbound symbols without specifying symbol=None

  • otp.Query now accepts an optional parameter params, which can contain a dictionary of parameters to be passed to the underlying query

Fixed#

  • query_inspector now properly considers symbol_param-dependent eval() queries as needing a lower bound (or unbound) symbol

[1.3.51]#

Fixed#

  • GeneratedDir might fail in the concurrent runs during the already exising directories

[1.3.50]#

Fixed#

  • otp.Custom with symbol as otp.query now returned merged data, not dict

[1.3.49]#

Added#

  • Add derived db support in the otp.DB

[1.3.48]#

Added#

  • Ability to add pins to a query

[1.3.47]#

Updated#

  • Usability of debugging scripts

[1.3.46]#

Updated#

  • Added pid to the base temp directory name to get rid of problems in multiprocess environment

[1.3.45]#

Updated#

  • Updated version of onetick.test

[1.3.44]#

Fixed#

  • removed debug output

[1.3.43]#

Fixed#

  • query_inspector now parses some old NESTED_OTQ formats properly

[1.3.42]#

Added#

  • logic that allows to integrate debugging logic for temporary generated objects

[1.3.41]#

Added#

  • query_inspector now can determine whether a query needs an unbound symbol list

[1.3.40]#

Fixed#

  • fixed bug related to the defining types using the getitem method

[1.3.39]#

Added#

  • specifying PYTHON_VERSION by default

[1.3.38]#

Added#

  • caching and prefix parameters to join_with_query

[1.3.37]#

Added#

  • Add symbols parameter to merge function to specify bound symbols for reading from db

  • otp.Custom now accept _Source object and collections (but not dicts) as symbols.

  • the otp.Empty source

  • ability to pass many databases in the Session.use method

Changed#

  • the keep_db flag set to False by default in the Symbols source

  • the join option in the join_with_query method to how, according to other join operations

[1.3.36]#

Added#

  • Implemented date difference in datepart units

[1.3.35]#

Fixed#

  • Added ability to use the Custom source with API binding using the ONE_TICK_CONFIG environment variable

[1.3.34]#

Fixed#

  • Time and TIMESTAMP columns type changed from the msectime to nsectime by default

[1.3.33]#

Fixed#

  • the query_inspector did not work with DECLARE_STATE_VARIABLES nodes, that has a different format that other EPs

[1.3.32]#

Added#

  • Check db name to be string

[1.3.31]#

Added#

  • join_with_query method to Source object

Deleted#

  • join_with_func function

Deprecated#

  • symbol method of Source object

[1.3.30]#

Fixed#

  • Earlier the Symbols returned symbols with a database prefix, but we introduce a keep_db flag and default value broke backward compatibility. Set default to True to keep compatibility

[1.3.29]#

Fixed#

  • Nested queries short path doesn’t work, when there is no otp.Session, but ONE_TICK_CONFIG is defined

[1.3.28]#

Fixed#

  • Issue related to the hardcoded name of input and output pins for the nested queries

Added#

  • query_inspector that reads a passed query and reveals a query structure

[1.3.27]#

Added#

  • Support external unbound symbols syntax

Fixed#

  • Bug of applying nested queries due the legacy passthrough

[1.3.26]#

Added#

  • otp.Year, otp.Quarter, …, otp.Nano classes can be used as parameter for + operation with timestamps

[1.3.25]#

Changed#

  • set onetick-lib version to 1.0.3 (works only with version of OneTick >= update3_20190927120000)

[1.3.24]#

Added#

  • regex support for drop method

[1.3.23]#

Fixed#

  • join_by_time didn’t assign name to a node in some conditions, that led to an error that one source didn’t have name specified

[1.3.22]#

Added#

  • check_schema flag to join_by_time for avoiding exception on joining Sources with columns’s names changed by sink event processor

[1.3.21]#

Fixed#

  • REGEX_EXTRACT supports case changing groups, while REGEX_REPLACE doesn’t

[1.3.20]#

Added#

  • automatically call rtrim on replace with column as a parameter

[1.3.19]#

Added#

  • DISTINCT aggregation

[1.3.18]#

Added#

  • .float accessor and str method to it

[1.3.17]#

Deprecated#

  • guess_schema argument of Custom constructor is deprecated

Added#

  • Custom constructor argument schema_policy (allowed values: manual, tolerant, fail) for better tuning of the schema guess behaviour

[1.3.16]#

Added#

  • str.substr function

[1.3.15]#

Added#

  • str.extract function

[1.3.14]#

Added#

  • str.repeat function and string by non-negative int multiplication based on this function

[1.3.13]#

Added#

  • str.find function

[1.3.12]#

Changed#

  • Moved locator_parser forward to 1.0.4

[1.3.11]#

Added#

  • str.replace and str.regex_replace functions

[1.3.10]#

Added#

  • utils omd_dist_path that finds OneTick build path bases on PYTHONPATH

[1.3.9]#

Added#

  • str.lower and str.upper functions

[1.3.8]#

Added#

  • str.trim, str.ltrim and str.rtrim functions

[1.3.7]#

Added#

  • columns generated by token saves parent dtype from now

[1.3.6]#

Added#

  • ability to change config file for a session

[1.3.5]#

Added#

  • str.len and str.contains functions

[1.3.4]#

Added#

  • Random seed setup at the beginning from session

[1.3.4]#

Added#

  • Random seed setup at the beginning from session

[1.3.3]#

Fixed#

  • Logging in temporary files that creates a lot of captured logs with pytest, that happens when logging file destroyed before temporary objects destruction

[1.3.2]#

Fixed#

  • Problem with overridden min/max functions: they missed kwargs params

[1.3.1]#

Updated#

  • Update pytest depenency from 4.x to 5.x

  • Update other pytest related dependencies

[1.2.54]#

Added#

  • Parameter shared_state_variables_list to function.apply_query

[1.2.53]#

Fixed#

  • Fixed bug with query params which had not propagated by apply_query method in the case if otp.sources.query.otp_query us as query parameter

[1.2.52]#

Fixed#

  • Partially defined DB locations. Also it starts to support optional location options like DAY_BOUNDARY_TZ

[1.2.51]#

Fixed#

  • Add ability to pass pd.Timestamp values, because it is know about DST. Also fixed a bug that did not allow to set datetime with finer precision than seconds.

[1.2.50]#

Fixed#

  • Added check for integration with onetick.query. Check raises a user friendly descriptive error message what to do. Also it cover a case when you had libraries for python with the standard malloc allocator in PYTHONPATH, but used python with the py-malloc allocator, then code rose magic problems on running queries but not on importing the onetick.query package.

[1.2.49]#

Fixed#

  • get_schema() and the like now support derived dbs properly

[1.2.48]#

Fixed#

  • remove pytest-profiling

[1.2.47]#

Added#

  • get_schema() set of utils; guess_schema parameter for Custom sources.

[1.2.46]#

Fixed#

  • Remove ALERTS database from the dummy locator

[1.2.45]#

Added#

  • Average aggregation; alternative (non-second) units for aggregations.

[1.2.44]#

Fixed#

  • Redirected OneTick logs into a temporary log file

[1.2.43]#

Fixed#

  • TmpFile does not take the confusing ‘path’ parameter

[1.2.42]#

Added#

  • Short version of conditions in lambdas and funcs, ie <> if row.x else <>

[1.2.41]#

Added#

  • Support min and max functions

[1.2.40]#

Added#

  • Partially support JWQ

[1.2.39]#

Added#

  • Support the lag operation on a column

[1.2.38]#

Fixed#

  • Remove quotes if they are passed when fill config file

[1.2.36]#

Added#

  • Fixed support lambdas for sources

[1.2.35]#

Added#

  • Added new accessor ‘match’ which is equivalent to ‘regex_match’ built-in function from OneTick.

[1.2.34]#

Added#

  • Mod operation support.

[1.2.33]#

Added#

  • “query” source config now supports dicts and lists as definitions of output columns.

[1.2.32]#

Added#

  • Support custom and server based license

[1.2.31]#

Fixed#

  • Assignment to timestamp field now applies a pre-sort.

[1.2.30]#

Added#

  • Added token accessor for string which call “token” built-in function from OneTick.

[1.2.29]#

Added#

  • Function get_schema() for getting the tick descriptors out of existing databases.

[1.2.28]#

Fixed#

  • Restored public interface for write_to_db.

[1.2.27]#

Added#

  • Support accessor for datetime and string

[1.2.26]#

Fixed#

  • Sessions are allowed to use COMMAND_EXECUTE EP.

[1.2.25]#

Fixed#

  • Fixed passing directory fixtures from the onetick.test packages to the Config constructor as otq_path and csv_path parameters without casting to string

[1.2.24]#

Added#

  • Support for join_by_time SAME_TIMESTAMP_JOIN_POLICY.

[1.2.23]#

Changed#

  • Interfaces for join and join_by_time functions.

  • Added merge function that repeats the concat functions

[1.2.22]#

Added#

  • Supported simple state variables

[1.2.21]#

Fixed#

  • Tests for ticks taking dates as input now work on Windows.

[1.2.20]#

Fixed#

  • Removing database from locator when some internal action was done (db.symbols, db.add …)

[1.2.19]#

Added#

  • Added property symbols to database object that returns all symbols in db.

  • Added source with symbols to get object we can work with

[1.2.18]#

Fixed#

  • Ability to add to db after session uses db

[1.2.17]#

Added#

  • Tick and Ticks objects can now be generated with datetime values as parameters, which will be cast to OneTick nanosecond-precision timestamps. (Module datetime itself doesn’t support ns time, so the initial values can only have microsecond precision.)

[1.2.16]#

Fixed#

  • CSV times are now read in proper (ns) resolution.

  • START/END timestamps are presented in proper (ns) resolution.

[1.2.15]#

Fixed#

  • to_otq query with eval in symbols

[1.2.14]#

Added#

  • Added symbol params support

[1.2.13]#

Added#

  • use_local_license flag to the Config; it is helpful when you don’t want to use local license and want to use remote license from a tick server

[1.2.12]#

Added#

  • Ability to add ‘destroy’ flag to acl for db

[1.2.11]#

Added#

  • It’s now possible to use pandas DataFrame’s as a source for databases.

[1.2.10]#

Added#

  • Ability to pass eval as an unbound symbol and otp.source.query as a symbol there

[1.2.9]#

Fixed#

  • Fixed the Ticks for case when columns has non-None values, but with different types

[1.2.8]#

Fixed#

  • to_graph() method to build the final graph instead of intermediate

[1.2.7.1]#

Added#

  • ‘empty’ flag to the locator, that allows to create a locator without stubs

[1.2.7]#

Added#

  • Restriction to have multiple active sessions simultaniously

[1.2.6]#

Added#

  • .split and .switch methods on the Source objects, that allows to split data. Cover SWITCH EP

[1.2.5]#

Added#

  • Ability to include reference to a locator in to a locator using. Locator().add(Locator(‘path/to/another.locator’)

  • Default values for a location in the DB constructor.

  • Allowed WRITE_TEXT in the default locator

[1.2.4]#

Changed#

  • Predefined databases became singletons instead of instances. Example: before was onetick.py.db.NYSE_TAQ(), but changed to onetick.py.db.NYSE_TAQ.

[1.2.3]#

Fixed#

  • Pre-defined db “NYSE_TAQ”

[1.2.2]#

Added#

  • Ability to add and remove “Admon user” to an ACL