Filtering#
Here are covered the basics of different filtering operations.
More complex “business” use-cases can be found in Filtering Use Cases
Let’s start with an unfiltered time series:
import onetick.py as otp
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
data = data[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
otp.run(data,
start=otp.dt(2024, 2, 1, 9, 30),
end=otp.dt(2024, 2, 1, 9, 30, 1),
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.000961260 | 184.010 | 302 | @FT | P |
| 1 | 2024-02-01 09:30:00.000961491 | 184.000 | 100 | @FT | P |
| 2 | 2024-02-01 09:30:00.000961701 | 184.000 | 1 | @FTI | P |
| 3 | 2024-02-01 09:30:00.000973163 | 184.000 | 1 | @FTI | P |
| 4 | 2024-02-01 09:30:00.000973355 | 184.000 | 5 | @FTI | P |
| ... | ... | ... | ... | ... | ... |
| 574 | 2024-02-01 09:30:00.987184691 | 183.900 | 9 | @F I | K |
| 575 | 2024-02-01 09:30:00.990378350 | 183.920 | 1 | @ I | D |
| 576 | 2024-02-01 09:30:00.991941892 | 183.935 | 1 | @ I | D |
| 577 | 2024-02-01 09:30:00.993785116 | 183.905 | 300 | @ | D |
| 578 | 2024-02-01 09:30:00.996512511 | 183.934 | 5 | @ I | D |
579 rows × 5 columns
Columns and Operations#
Simple filtering can be expressed by comparing the values of columns
or the results of operations that use columns.
Method where() can be used to filter data
with operations that return boolean result (True or False).
For example, we can compare the value of the field
to some string literal or number with == operator:
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
data = data[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
# get only the ticks with EXCHANGE field equal to K
data = data.where(data['EXCHANGE'] == 'K')
otp.run(data,
start=otp.dt(2024, 2, 1, 9, 30),
end=otp.dt(2024, 2, 1, 9, 30, 1),
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.080154806 | 183.95 | 1 | @ I | K |
| 1 | 2024-02-01 09:30:00.080155543 | 183.95 | 5 | @ I | K |
| 2 | 2024-02-01 09:30:00.080534432 | 183.90 | 2 | @ I | K |
| 3 | 2024-02-01 09:30:00.080893236 | 183.97 | 15 | @F I | K |
| 4 | 2024-02-01 09:30:00.080984540 | 183.90 | 86 | @ I | K |
| ... | ... | ... | ... | ... | ... |
| 66 | 2024-02-01 09:30:00.954126905 | 183.92 | 2 | @F I | K |
| 67 | 2024-02-01 09:30:00.954404272 | 183.92 | 5 | @F I | K |
| 68 | 2024-02-01 09:30:00.954697550 | 183.92 | 2 | @F I | K |
| 69 | 2024-02-01 09:30:00.985733751 | 183.90 | 19 | @F I | K |
| 70 | 2024-02-01 09:30:00.987184691 | 183.90 | 9 | @F I | K |
71 rows × 5 columns
Note that all of the filtering is done in OneTick not in Python, which is much more efficient and lets us work with much bigger data sets.
Binary operations#
Filters can also include and/or binary logic
with & and | operators:
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
data = data[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
# get only the ticks which have both EXCHANGE field equal to K and the PRICE bigger than 183.950
data = data.where((data['EXCHANGE'] == 'K') & (data['PRICE'] > 183.950))
otp.run(data,
start=otp.dt(2024, 2, 1, 9, 30),
end=otp.dt(2024, 2, 1, 9, 30, 1),
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.080893236 | 183.97 | 15 | @F I | K |
| 1 | 2024-02-01 09:30:00.153161386 | 183.98 | 50 | @F I | K |
| 2 | 2024-02-01 09:30:00.153953889 | 183.98 | 29 | @F I | K |
| 3 | 2024-02-01 09:30:00.154413668 | 183.98 | 21 | @F I | K |
| 4 | 2024-02-01 09:30:00.155845868 | 183.98 | 50 | @F I | K |
| ... | ... | ... | ... | ... | ... |
| 13 | 2024-02-01 09:30:00.614422140 | 183.96 | 25 | @F I | K |
| 14 | 2024-02-01 09:30:00.621498637 | 183.96 | 52 | @ I | K |
| 15 | 2024-02-01 09:30:00.621587688 | 183.96 | 382 | @ | K |
| 16 | 2024-02-01 09:30:00.623837515 | 183.96 | 25 | @ I | K |
| 17 | 2024-02-01 09:30:00.623851984 | 183.96 | 1 | @ I | K |
18 rows × 5 columns
String operations#
Many methods are available on a string columns with .str accessor.
For example, string search can be performed with regular expressions using
.str.match
or with SQL like expressions using
.str.ilike.
Filtering for a specific trade condition can be done with
.str.contains:
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
data = data[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
# get only the ticks which have I character in the COND field
data = data.where(data['COND'].str.contains('I'))
otp.run(data,
start=otp.dt(2024, 2, 1, 9, 30),
end=otp.dt(2024, 2, 1, 9, 30, 1),
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.000961701 | 184.000 | 1 | @FTI | P |
| 1 | 2024-02-01 09:30:00.000973163 | 184.000 | 1 | @FTI | P |
| 2 | 2024-02-01 09:30:00.000973355 | 184.000 | 5 | @FTI | P |
| 3 | 2024-02-01 09:30:00.000973517 | 184.000 | 5 | @FTI | P |
| 4 | 2024-02-01 09:30:00.000973674 | 184.000 | 5 | @FTI | P |
| ... | ... | ... | ... | ... | ... |
| 464 | 2024-02-01 09:30:00.985733751 | 183.900 | 19 | @F I | K |
| 465 | 2024-02-01 09:30:00.987184691 | 183.900 | 9 | @F I | K |
| 466 | 2024-02-01 09:30:00.990378350 | 183.920 | 1 | @ I | D |
| 467 | 2024-02-01 09:30:00.991941892 | 183.935 | 1 | @ I | D |
| 468 | 2024-02-01 09:30:00.996512511 | 183.934 | 5 | @ I | D |
469 rows × 5 columns
Different sets of methods are available for .dt
and .float accessors.
Float comparison#
Because of the way float values are stored in the memory,
you can have some unprecise results when comparing them with == operator.
Instead it’s recommended to use
.float.cmp
or .float.eq methods,
which allow to specify precision of comparison manually:
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
data = data[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
# get only the ticks where PRICE is bigger than 183.950 with 0.0000001 relative difference
data = data.where(data['PRICE'].float.cmp(183.950, 0.0000001) == 1)
otp.run(data,
start=otp.dt(2024, 2, 1, 9, 30),
end=otp.dt(2024, 2, 1, 9, 30, 1),
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.000961260 | 184.010 | 302 | @FT | P |
| 1 | 2024-02-01 09:30:00.000961491 | 184.000 | 100 | @FT | P |
| 2 | 2024-02-01 09:30:00.000961701 | 184.000 | 1 | @FTI | P |
| 3 | 2024-02-01 09:30:00.000973163 | 184.000 | 1 | @FTI | P |
| 4 | 2024-02-01 09:30:00.000973355 | 184.000 | 5 | @FTI | P |
| ... | ... | ... | ... | ... | ... |
| 224 | 2024-02-01 09:30:00.760947045 | 183.960 | 2 | @F I | B |
| 225 | 2024-02-01 09:30:00.780798794 | 183.960 | 5 | @ I | B |
| 226 | 2024-02-01 09:30:00.781988281 | 183.980 | 1 | @ I | D |
| 227 | 2024-02-01 09:30:00.791429521 | 184.005 | 1 | @ I | D |
| 228 | 2024-02-01 09:30:00.970872668 | 183.980 | 1 | @ I | D |
229 rows × 5 columns
Also there are otp.nan and otp.inf special objects in OneTick
that can be used to filter out NaN and Infinite values.
In OneTick they can be safely used with == operator,
unlike common implementations which treat comparison with NaN as always returning False.
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
# calculate hourly VWAP, then filter out NaN buckets
data = data.agg({'VWAP': otp.agg.vwap('PRICE', 'SIZE')}, bucket_interval=otp.Hour(1))
data = data.where(data['VWAP'] != otp.nan)
otp.run(data,
start=otp.dt(2024, 2, 1, 0, 0),
end=otp.dt(2024, 2, 1, 10, 0),
timezone='America/New_York',
symbols='AAPL')
| Time | VWAP | |
|---|---|---|
| 0 | 2024-02-01 05:00:00 | 185.685111 |
| 1 | 2024-02-01 06:00:00 | 185.444859 |
| 2 | 2024-02-01 07:00:00 | 185.478817 |
| 3 | 2024-02-01 08:00:00 | 185.613162 |
| 4 | 2024-02-01 09:00:00 | 184.704087 |
| 5 | 2024-02-01 10:00:00 | 184.721101 |
Filtering methods#
Source class also have many other methods that work as a filter.
For example, filter that limits attention to on-exchange continuous trading trades
can be implemented like this with character_present() method:
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
data = data[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
# get only the ticks where COND field doesn't contain specified characters
data = data.character_present(data['COND'], 'O6TUHILNRWZ47QMBCGPV', discard_on_match=True)
otp.run(data,
start=otp.dt(2024, 2, 1, 9, 30),
end=otp.dt(2024, 2, 1, 9, 30, 1),
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.080096628 | 183.985 | 154 | @F | P |
| 1 | 2024-02-01 09:30:00.080109023 | 183.900 | 150 | @F | P |
| 2 | 2024-02-01 09:30:00.080109175 | 183.900 | 477 | @F | P |
| 3 | 2024-02-01 09:30:00.080904251 | 183.900 | 523 | @F | P |
| 4 | 2024-02-01 09:30:00.081016071 | 183.900 | 100 | @ | P |
| ... | ... | ... | ... | ... | ... |
| 80 | 2024-02-01 09:30:00.795482321 | 183.945 | 100 | @ | D |
| 81 | 2024-02-01 09:30:00.796118938 | 183.920 | 100 | @F | U |
| 82 | 2024-02-01 09:30:00.833291631 | 183.910 | 100 | @F | Q |
| 83 | 2024-02-01 09:30:00.945031779 | 183.920 | 100 | @ | K |
| 84 | 2024-02-01 09:30:00.993785116 | 183.905 | 300 | @ | D |
85 rows × 5 columns
Filtering out outliers or missing values#
Method skip_bad_tick() can be used to filter out the ticks which have field values that
differ too much from the values of the same fields in the surrounding ticks.
Method dropna() can be used to filter out the ticks with NaN values in the fields,
similar to how we do it in the “Float comparison” section.
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
data = data[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
# skip bad PRICEs
data = data.skip_bad_tick('PRICE')
# drop all ticks with NaN values in any field
data = data.dropna()
otp.run(data,
start=otp.dt(2024, 2, 1, 9, 30),
end=otp.dt(2024, 2, 1, 9, 30, 1),
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.000961260 | 184.010 | 302 | @FT | P |
| 1 | 2024-02-01 09:30:00.000961491 | 184.000 | 100 | @FT | P |
| 2 | 2024-02-01 09:30:00.000961701 | 184.000 | 1 | @FTI | P |
| 3 | 2024-02-01 09:30:00.000973163 | 184.000 | 1 | @FTI | P |
| 4 | 2024-02-01 09:30:00.000973355 | 184.000 | 5 | @FTI | P |
| ... | ... | ... | ... | ... | ... |
| 574 | 2024-02-01 09:30:00.987184691 | 183.900 | 9 | @F I | K |
| 575 | 2024-02-01 09:30:00.990378350 | 183.920 | 1 | @ I | D |
| 576 | 2024-02-01 09:30:00.991941892 | 183.935 | 1 | @ I | D |
| 577 | 2024-02-01 09:30:00.993785116 | 183.905 | 300 | @ | D |
| 578 | 2024-02-01 09:30:00.996512511 | 183.934 | 5 | @ I | D |
579 rows × 5 columns
Time filtering#
Ticks can be filtered in OneTick by setting the query time range or filtering the ticks by their timestamps.
Setting the time range of the query#
Query time range can be set in otp.run function with start and end parameters,
as was done in all previous examples.
Also these parameters can instead be set on
otp.DataSource object (and some other sources):
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD',
start=otp.dt(2024, 2, 1, 9, 30),
end=otp.dt(2024, 2, 1, 9, 30, 1))
data = data[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
otp.run(data,
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.000961260 | 184.010 | 302 | @FT | P |
| 1 | 2024-02-01 09:30:00.000961491 | 184.000 | 100 | @FT | P |
| 2 | 2024-02-01 09:30:00.000961701 | 184.000 | 1 | @FTI | P |
| 3 | 2024-02-01 09:30:00.000973163 | 184.000 | 1 | @FTI | P |
| 4 | 2024-02-01 09:30:00.000973355 | 184.000 | 5 | @FTI | P |
| ... | ... | ... | ... | ... | ... |
| 574 | 2024-02-01 09:30:00.987184691 | 183.900 | 9 | @F I | K |
| 575 | 2024-02-01 09:30:00.990378350 | 183.920 | 1 | @ I | D |
| 576 | 2024-02-01 09:30:00.991941892 | 183.935 | 1 | @ I | D |
| 577 | 2024-02-01 09:30:00.993785116 | 183.905 | 300 | @ | D |
| 578 | 2024-02-01 09:30:00.996512511 | 183.934 | 5 | @ I | D |
579 rows × 5 columns
Apply times daily#
Parameter apply_times_daily in otp.run function can be used
to apply the same time range for each day when querying several days:
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
data = data[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
# in this case time range will be set from 09:30:00 to 09:30:00.002
# for days 2024-02-01 and 2024-02-02
otp.run(data,
start=otp.dt(2024, 2, 1, 9, 30, 0),
end=otp.dt(2024, 2, 2, 9, 30, 0, 2000),
timezone='America/New_York',
apply_times_daily=True,
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.000961260 | 184.01 | 302 | @FT | P |
| 1 | 2024-02-01 09:30:00.000961491 | 184.00 | 100 | @FT | P |
| 2 | 2024-02-01 09:30:00.000961701 | 184.00 | 1 | @FTI | P |
| 3 | 2024-02-01 09:30:00.000973163 | 184.00 | 1 | @FTI | P |
| 4 | 2024-02-01 09:30:00.000973355 | 184.00 | 5 | @FTI | P |
| ... | ... | ... | ... | ... | ... |
| 15 | 2024-02-01 09:30:00.000996695 | 184.00 | 10 | @FTI | P |
| 16 | 2024-02-01 09:30:00.000999752 | 184.00 | 100 | @FT | P |
| 17 | 2024-02-01 09:30:00.001730949 | 183.91 | 50 | @FTI | Q |
| 18 | 2024-02-02 09:30:00.001403754 | 179.85 | 1 | @FTI | Q |
| 19 | 2024-02-02 09:30:00.001416711 | 179.87 | 228 | @FT | Q |
20 rows × 5 columns
Using time filtering methods and operations#
Similar methods as in previous examples can be used to filter ticks by their timestamps:
TIMESTAMPmeta field comparisontime_filter()method
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
data = data[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
# filter the ticks inside the 9:30-16:00 time range for each day
data = data.time_filter(start_time='093000000', end_time='160000000')
# filter ticks before 2024-02-01 09:30:01
# (note that this is a fixed datetime value
# and won't work on a daily basis like apply_times_daily or time_filter)
data = data.where(data['TIMESTAMP'] < otp.dt(2024, 2, 1, 9, 30, 1))
otp.run(data,
start=otp.dt(2024, 2, 1),
end=otp.dt(2024, 2, 3),
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.000961260 | 184.010 | 302 | @FT | P |
| 1 | 2024-02-01 09:30:00.000961491 | 184.000 | 100 | @FT | P |
| 2 | 2024-02-01 09:30:00.000961701 | 184.000 | 1 | @FTI | P |
| 3 | 2024-02-01 09:30:00.000973163 | 184.000 | 1 | @FTI | P |
| 4 | 2024-02-01 09:30:00.000973355 | 184.000 | 5 | @FTI | P |
| ... | ... | ... | ... | ... | ... |
| 574 | 2024-02-01 09:30:00.987184691 | 183.900 | 9 | @F I | K |
| 575 | 2024-02-01 09:30:00.990378350 | 183.920 | 1 | @ I | D |
| 576 | 2024-02-01 09:30:00.991941892 | 183.935 | 1 | @ I | D |
| 577 | 2024-02-01 09:30:00.993785116 | 183.905 | 300 | @ | D |
| 578 | 2024-02-01 09:30:00.996512511 | 183.934 | 5 | @ I | D |
579 rows × 5 columns
Getting first and last ticks#
We can limit the number of ticks returned with the help of limit() method
(or using first() and last() aggregations):
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
data = data[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
# get first 100 ticks
data = data.limit(100)
otp.run(data,
start=otp.dt(2024, 2, 1, 9, 30),
end=otp.dt(2024, 2, 1, 9, 30, 1),
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.000961260 | 184.01 | 302 | @FT | P |
| 1 | 2024-02-01 09:30:00.000961491 | 184.00 | 100 | @FT | P |
| 2 | 2024-02-01 09:30:00.000961701 | 184.00 | 1 | @FTI | P |
| 3 | 2024-02-01 09:30:00.000973163 | 184.00 | 1 | @FTI | P |
| 4 | 2024-02-01 09:30:00.000973355 | 184.00 | 5 | @FTI | P |
| ... | ... | ... | ... | ... | ... |
| 95 | 2024-02-01 09:30:00.152501252 | 183.98 | 16 | @ I | P |
| 96 | 2024-02-01 09:30:00.152623426 | 183.97 | 10 | @F I | U |
| 97 | 2024-02-01 09:30:00.152623523 | 183.97 | 15 | @F I | U |
| 98 | 2024-02-01 09:30:00.153016032 | 183.97 | 10 | @F I | U |
| 99 | 2024-02-01 09:30:00.153153673 | 183.96 | 35 | @F I | Z |
100 rows × 5 columns
Getting last 5 ticks from the previous example:
# get last 5 ticks
data = data.last(5)
otp.run(data,
start=otp.dt(2024, 2, 1, 9, 30),
end=otp.dt(2024, 2, 1, 9, 30, 1),
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.152501252 | 183.98 | 16 | @ I | P |
| 1 | 2024-02-01 09:30:00.152623426 | 183.97 | 10 | @F I | U |
| 2 | 2024-02-01 09:30:00.152623523 | 183.97 | 15 | @F I | U |
| 3 | 2024-02-01 09:30:00.153016032 | 183.97 | 10 | @F I | U |
| 4 | 2024-02-01 09:30:00.153153673 | 183.96 | 35 | @F I | Z |
Slice syntax#
For limiting the ticks Python-like slice syntax is also supported with __getitem__():
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
data = data[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
# get last 100 ticks
data = data[-100:]
otp.run(data,
start=otp.dt(2024, 2, 1, 9, 30),
end=otp.dt(2024, 2, 1, 9, 30, 1),
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.709007667 | 183.960 | 1 | @ I | P |
| 1 | 2024-02-01 09:30:00.712853192 | 183.980 | 1 | @ I | D |
| 2 | 2024-02-01 09:30:00.716359511 | 183.960 | 24 | @F I | P |
| 3 | 2024-02-01 09:30:00.716359612 | 183.960 | 100 | @F | P |
| 4 | 2024-02-01 09:30:00.716359707 | 183.960 | 50 | @F I | P |
| ... | ... | ... | ... | ... | ... |
| 95 | 2024-02-01 09:30:00.987184691 | 183.900 | 9 | @F I | K |
| 96 | 2024-02-01 09:30:00.990378350 | 183.920 | 1 | @ I | D |
| 97 | 2024-02-01 09:30:00.991941892 | 183.935 | 1 | @ I | D |
| 98 | 2024-02-01 09:30:00.993785116 | 183.905 | 300 | @ | D |
| 99 | 2024-02-01 09:30:00.996512511 | 183.934 | 5 | @ I | D |
100 rows × 5 columns
Filtering with aggregations#
Some aggregations can be used as filters too.
For example, using parameter group_by in first() aggregation
will output the first tick for each group:
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
data = data[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
# get first tick for each exchange
data = data.first(group_by='EXCHANGE')
otp.run(data,
start=otp.dt(2024, 2, 1, 9, 30),
end=otp.dt(2024, 2, 1, 9, 30, 1),
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.000961260 | 184.010 | 302 | @FT | P |
| 1 | 2024-02-01 09:30:00.001730949 | 183.910 | 50 | @FTI | Q |
| 2 | 2024-02-01 09:30:00.080154806 | 183.950 | 1 | @ I | K |
| 3 | 2024-02-01 09:30:00.080519001 | 183.960 | 1 | @F I | Z |
| 4 | 2024-02-01 09:30:00.080622883 | 183.970 | 86 | @F I | Y |
| ... | ... | ... | ... | ... | ... |
| 8 | 2024-02-01 09:30:00.081022425 | 183.980 | 1 | @F I | H |
| 9 | 2024-02-01 09:30:00.081124084 | 183.980 | 1 | @F I | N |
| 10 | 2024-02-01 09:30:00.147383589 | 183.985 | 5 | @ I | V |
| 11 | 2024-02-01 09:30:00.429641068 | 183.985 | 1 | @ I | D |
| 12 | 2024-02-01 09:30:00.496570015 | 183.960 | 15 | @ I | B |
13 rows × 5 columns
Splitting the data with filter#
We can also return two branches after filtering with method where_clause()
(or __getitem__()).
The first branch contains all ticks that satisfy the condition:
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
data = data[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
# split the data flow into two branches
exchange_k, other = data[data['EXCHANGE'] == 'K']
otp.run(exchange_k,
start=otp.dt(2024, 2, 1, 9, 30),
end=otp.dt(2024, 2, 1, 9, 30, 1),
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.080154806 | 183.95 | 1 | @ I | K |
| 1 | 2024-02-01 09:30:00.080155543 | 183.95 | 5 | @ I | K |
| 2 | 2024-02-01 09:30:00.080534432 | 183.90 | 2 | @ I | K |
| 3 | 2024-02-01 09:30:00.080893236 | 183.97 | 15 | @F I | K |
| 4 | 2024-02-01 09:30:00.080984540 | 183.90 | 86 | @ I | K |
| ... | ... | ... | ... | ... | ... |
| 66 | 2024-02-01 09:30:00.954126905 | 183.92 | 2 | @F I | K |
| 67 | 2024-02-01 09:30:00.954404272 | 183.92 | 5 | @F I | K |
| 68 | 2024-02-01 09:30:00.954697550 | 183.92 | 2 | @F I | K |
| 69 | 2024-02-01 09:30:00.985733751 | 183.90 | 19 | @F I | K |
| 70 | 2024-02-01 09:30:00.987184691 | 183.90 | 9 | @F I | K |
71 rows × 5 columns
And the other branch contains all ticks that do not satisfy the condition:
otp.run(other,
start=otp.dt(2024, 2, 1, 9, 30),
end=otp.dt(2024, 2, 1, 9, 30, 1),
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.000961260 | 184.010 | 302 | @FT | P |
| 1 | 2024-02-01 09:30:00.000961491 | 184.000 | 100 | @FT | P |
| 2 | 2024-02-01 09:30:00.000961701 | 184.000 | 1 | @FTI | P |
| 3 | 2024-02-01 09:30:00.000973163 | 184.000 | 1 | @FTI | P |
| 4 | 2024-02-01 09:30:00.000973355 | 184.000 | 5 | @FTI | P |
| ... | ... | ... | ... | ... | ... |
| 503 | 2024-02-01 09:30:00.972086329 | 183.935 | 1 | @ I | D |
| 504 | 2024-02-01 09:30:00.990378350 | 183.920 | 1 | @ I | D |
| 505 | 2024-02-01 09:30:00.991941892 | 183.935 | 1 | @ I | D |
| 506 | 2024-02-01 09:30:00.993785116 | 183.905 | 300 | @ | D |
| 507 | 2024-02-01 09:30:00.996512511 | 183.934 | 5 | @ I | D |
508 rows × 5 columns
Creating filters from another query#
It is possible to construct filter expression dynamically
as a result of another query using otp.eval.
For example, we can filter ticks with the PRICE bigger than the median price for the day:
data = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
data = data[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
# calculating median price for the day
median = data.agg({'MEDIAN': otp.agg.median('PRICE')})
median['WHERE'] = 'PRICE > ' + median['MEDIAN'].astype(str)
# and using it as a filter
data = data.where(otp.eval(median[['WHERE']]))
otp.run(data,
start=otp.dt(2024, 2, 1, 9, 30),
end=otp.dt(2024, 2, 1, 16, 0),
timezone='America/New_York',
symbols='AAPL')
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 10:07:18.700705838 | 185.9200 | 40 | @ I | D |
| 1 | 2024-02-01 10:07:18.778908902 | 185.9150 | 50 | @ I | D |
| 2 | 2024-02-01 10:07:25.972610558 | 185.9181 | 1 | @ I | D |
| 3 | 2024-02-01 10:07:25.992016855 | 185.9192 | 1 | @ I | D |
| 4 | 2024-02-01 10:07:26.045516588 | 185.9150 | 533 | @ | D |
| ... | ... | ... | ... | ... | ... |
| 332778 | 2024-02-01 15:59:59.976861489 | 186.8388 | 1 | @ I | D |
| 332779 | 2024-02-01 15:59:59.990154615 | 186.8900 | 96 | @F I | Q |
| 332780 | 2024-02-01 15:59:59.990157934 | 186.8900 | 100 | @F | Q |
| 332781 | 2024-02-01 15:59:59.990260269 | 186.8900 | 8 | @F I | U |
| 332782 | 2024-02-01 15:59:59.990606937 | 186.8312 | 1 | @ I | D |
332783 rows × 5 columns