Variables and Data Structures#
Variables (aka ‘state variables’)#
Variables can be used to keep track of state across ticks. The example below shows how we may keep track of P&L.
import onetick.py as otp
s = otp.dt(2024, 2, 1, 9, 30)
e = otp.dt(2024, 2, 1, 9, 30, 1)
trd = otp.Ticks({'PRICE': [13.5, 13.6, 13.3, 14.0],
'SIZE': [200, 100, 150, 200],
'SIDE': ['B', 'S', 'B', 'S']})
trd.state_vars['PROFIT'] = 0
trd.state_vars['PROFIT'] += trd.apply(
lambda t: t['PRICE'] * t['SIZE'] if t['SIDE'] == 'S' else -(t['PRICE'] * t['SIZE'])
)
trd['PROFIT'] = trd.state_vars['PROFIT']
otp.run(trd)
| Time | PRICE | SIZE | SIDE | PROFIT | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 00:00:00.000 | 13.5 | 200 | B | -2700 |
| 1 | 2024-02-01 00:00:00.001 | 13.6 | 100 | S | -1340 |
| 2 | 2024-02-01 00:00:00.002 | 13.3 | 150 | B | -3335 |
| 3 | 2024-02-01 00:00:00.003 | 14.0 | 200 | S | -535 |
The variable ‘PROFIT’ keeps a running total. In other words, it aggregates state across trades.
Note that the same can be accomplished without variables by keeping the running total in a separate column.
trd = otp.Ticks({'PRICE': [13.5, 13.6, 13.3, 14.0],
'SIZE': [200, 100, 150, 200],
'SIDE': ['B', 'S', 'B', 'S']})
trd['VALUE'] = trd.apply(
lambda t: t['PRICE'] * t['SIZE'] if t['SIDE'] == 'S' else -(t['PRICE'] * t['SIZE'])
)
trd = trd.agg({'PROFIT': otp.agg.sum('VALUE')}, running=True)
otp.run(trd)
| Time | PROFIT | |
|---|---|---|
| 0 | 2024-02-01 00:00:00.000 | -2700.0 |
| 1 | 2024-02-01 00:00:00.001 | -1340.0 |
| 2 | 2024-02-01 00:00:00.002 | -3335.0 |
| 3 | 2024-02-01 00:00:00.003 | -535.0 |
Below is an example of using variables that cannot be as easily implemented with running totals: remembering the value of the last tick and referring to it after grouping/aggregation.
q = otp.Ticks(X=[-1, 3, -3, 4, 2], Y=[0, 1, 1, 0, 3])
q.state_vars['last_x'] = 0
q.state_vars['last_x'] = q['X']
q = q.high('X', group_by=['Y'])
q['last_x'] = q.state_vars['last_x']
otp.run(q)
| Time | X | Y | last_x | |
|---|---|---|---|---|
| 0 | 2024-02-01 00:00:00.001 | 3 | 1 | 2 |
| 1 | 2024-02-01 00:00:00.003 | 4 | 0 | 2 |
| 2 | 2024-02-01 00:00:00.004 | 2 | 3 | 2 |
Dictionaries / Maps (aka tick sets)#
Looking up static data for every tick#
A map can be created with keys taken from one or more columns and holding entire ticks as values.
The example below uses exchange reference data to create a map keyed by exchange code.
exchanges = otp.Ticks(
EXCHANGE=['A', 'B', 'C', 'D', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'P', 'Q', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'],
NAME=['NYSE American (Amex)', 'Nasdaq BX', 'NYSE National (NSX)', 'FINRA ADF + NYSE/Nasdaq TRFs', 'MIAX Pearl', 'International Securities Exchange', 'Cboe EDGA', 'Cboe EDGX', 'Long-Term Stock Exchange (LTSE)', 'NYSE Chicago', 'New York Stock Exchange', 'NYSE Arca', 'Nasdaq (Tape C securities)', 'Consolidated Tape System (CTS)', 'Nasdaq (Tape A,B securities)', 'Members Exchange (MEMX)', "The Investors' Exchange (IEX)", 'CBOE Stock Exchange (CBSX)', 'Nasdaq PSX', 'Cboe BYX', 'Cboe BZX']
)
exchanges['LOCATION'] = 'US'
otp.run(exchanges)
| Time | EXCHANGE | NAME | LOCATION | |
|---|---|---|---|---|
| 0 | 2024-02-01 00:00:00.000 | A | NYSE American (Amex) | US |
| 1 | 2024-02-01 00:00:00.001 | B | Nasdaq BX | US |
| 2 | 2024-02-01 00:00:00.002 | C | NYSE National (NSX) | US |
| 3 | 2024-02-01 00:00:00.003 | D | FINRA ADF + NYSE/Nasdaq TRFs | US |
| 4 | 2024-02-01 00:00:00.004 | H | MIAX Pearl | US |
| ... | ... | ... | ... | ... |
| 16 | 2024-02-01 00:00:00.016 | V | The Investors' Exchange (IEX) | US |
| 17 | 2024-02-01 00:00:00.017 | W | CBOE Stock Exchange (CBSX) | US |
| 18 | 2024-02-01 00:00:00.018 | X | Nasdaq PSX | US |
| 19 | 2024-02-01 00:00:00.019 | Y | Cboe BYX | US |
| 20 | 2024-02-01 00:00:00.020 | Z | Cboe BZX | US |
21 rows × 4 columns
We will add exchange name to trades. First let’s examine the trades.
q = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
q = q[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
otp.run(q, start=s, end=e, symbols=['AAPL'])
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.000961260 | 184.010 | 302 | @FT | P |
| 1 | 2024-02-01 09:30:00.000961491 | 184.000 | 100 | @FT | P |
| 2 | 2024-02-01 09:30:00.000961701 | 184.000 | 1 | @FTI | P |
| 3 | 2024-02-01 09:30:00.000973163 | 184.000 | 1 | @FTI | P |
| 4 | 2024-02-01 09:30:00.000973355 | 184.000 | 5 | @FTI | P |
| ... | ... | ... | ... | ... | ... |
| 574 | 2024-02-01 09:30:00.987184691 | 183.900 | 9 | @F I | K |
| 575 | 2024-02-01 09:30:00.990378350 | 183.920 | 1 | @ I | D |
| 576 | 2024-02-01 09:30:00.991941892 | 183.935 | 1 | @ I | D |
| 577 | 2024-02-01 09:30:00.993785116 | 183.905 | 300 | @ | D |
| 578 | 2024-02-01 09:30:00.996512511 | 183.934 | 5 | @ I | D |
579 rows × 5 columns
The value of the EXCHANGE field from trades will be used to look up the name of the corresponding exchange.
q = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
q = q[['PRICE','SIZE','COND','EXCHANGE']]
exchanges = otp.Ticks(
EXCHANGE=['A', 'B', 'C', 'D', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'P', 'Q', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'],
NAME=['NYSE American (Amex)', 'Nasdaq BX', 'NYSE National (NSX)', 'FINRA ADF + NYSE/Nasdaq TRFs', 'MIAX Pearl', 'International Securities Exchange', 'Cboe EDGA', 'Cboe EDGX', 'Long-Term Stock Exchange (LTSE)', 'NYSE Chicago', 'New York Stock Exchange', 'NYSE Arca', 'Nasdaq (Tape C securities)', 'Consolidated Tape System (CTS)', 'Nasdaq (Tape A,B securities)', 'Members Exchange (MEMX)', "The Investors' Exchange (IEX)", 'CBOE Stock Exchange (CBSX)', 'Nasdaq PSX', 'Cboe BYX', 'Cboe BZX']
)
q.state_vars['exchanges'] = otp.state.tick_set('latest', 'EXCHANGE', otp.eval(exchanges))
q['exchange_name'] = q.state_vars['exchanges'].find('NAME', 'unknown')
otp.run(q, start=s, end=e, symbols=['AAPL'])
| Time | PRICE | SIZE | COND | EXCHANGE | exchange_name | |
|---|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.000961260 | 184.010 | 302 | @FT | P | NYSE Arca |
| 1 | 2024-02-01 09:30:00.000961491 | 184.000 | 100 | @FT | P | NYSE Arca |
| 2 | 2024-02-01 09:30:00.000961701 | 184.000 | 1 | @FTI | P | NYSE Arca |
| 3 | 2024-02-01 09:30:00.000973163 | 184.000 | 1 | @FTI | P | NYSE Arca |
| 4 | 2024-02-01 09:30:00.000973355 | 184.000 | 5 | @FTI | P | NYSE Arca |
| ... | ... | ... | ... | ... | ... | ... |
| 574 | 2024-02-01 09:30:00.987184691 | 183.900 | 9 | @F I | K | Cboe EDGX |
| 575 | 2024-02-01 09:30:00.990378350 | 183.920 | 1 | @ I | D | FINRA ADF + NYSE/Nasdaq TRFs |
| 576 | 2024-02-01 09:30:00.991941892 | 183.935 | 1 | @ I | D | FINRA ADF + NYSE/Nasdaq TRFs |
| 577 | 2024-02-01 09:30:00.993785116 | 183.905 | 300 | @ | D | FINRA ADF + NYSE/Nasdaq TRFs |
| 578 | 2024-02-01 09:30:00.996512511 | 183.934 | 5 | @ I | D | FINRA ADF + NYSE/Nasdaq TRFs |
579 rows × 6 columns
Checking if ticks are present in another time series#
A tick set can be used to check if (a small number of) ticks are present in another time series with potentially different time stamps.
We generate two “special” ticks.
ticks = otp.Ticks(PRICE=[184, 184], SIZE=[1, 2], INDEX=[1, 2])
otp.run(ticks)
| Time | PRICE | SIZE | INDEX | |
|---|---|---|---|---|
| 0 | 2024-02-01 00:00:00.000 | 184 | 1 | 1 |
| 1 | 2024-02-01 00:00:00.001 | 184 | 2 | 2 |
We then check if any of the trades match these ticks by PRICE and SIZE.
q = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD')
q = q[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
q.state_vars['special_ticks'] = otp.state.tick_set('latest', ['PRICE', 'SIZE'], otp.eval(ticks))
q['SPECIAL'] = q.state_vars['special_ticks'].find('INDEX', -1)
q = q.where(q['SPECIAL'] != -1)
otp.run(q, start=s, end=e, symbols=['AAPL'])
| Time | PRICE | SIZE | COND | EXCHANGE | SPECIAL | |
|---|---|---|---|---|---|---|
| 0 | 2024-02-01 09:30:00.000961701 | 184.0 | 1 | @FTI | P | 1 |
| 1 | 2024-02-01 09:30:00.000973163 | 184.0 | 1 | @FTI | P | 1 |
| 2 | 2024-02-01 09:30:00.000980967 | 184.0 | 1 | @FTI | P | 1 |
| 3 | 2024-02-01 09:30:00.000984626 | 184.0 | 1 | @FTI | P | 1 |
| 4 | 2024-02-01 09:30:00.000984730 | 184.0 | 1 | @FTI | P | 1 |
| ... | ... | ... | ... | ... | ... | ... |
| 7 | 2024-02-01 09:30:00.000994257 | 184.0 | 2 | @FTI | P | 2 |
| 8 | 2024-02-01 09:30:00.277375099 | 184.0 | 2 | @ I | K | 2 |
| 9 | 2024-02-01 09:30:00.384757464 | 184.0 | 1 | @ I | Z | 1 |
| 10 | 2024-02-01 09:30:00.625020981 | 184.0 | 1 | @ I | D | 1 |
| 11 | 2024-02-01 09:30:00.632118285 | 184.0 | 1 | @ I | D | 1 |
12 rows × 6 columns
Lists and Queues#
Lists and double-ended queues (deques) can be used to keep track of collections ticks.