Variables and Data Structures#
Variables (aka ‘state variables’)#
Variables can be used to keep track of state across ticks. The example below shows how we may keep track of P&L.
import onetick.py as otp
s = otp.dt(2023, 5, 15, 9, 30)
e = otp.dt(2023, 5, 15, 9, 30, 1)
trd = otp.Ticks({'PRICE': [13.5, 13.6, 13.3, 14.0],
'SIZE': [200, 100, 150, 200],
'SIDE': ['B', 'S', 'B', 'S']})
trd.state_vars['PROFIT'] = 0
trd.state_vars['PROFIT'] += trd.apply(
lambda t: t['PRICE'] * t['SIZE'] if t['SIDE'] == 'S' else -(t['PRICE'] * t['SIZE'])
)
trd['PROFIT'] = trd.state_vars['PROFIT']
otp.run(trd)
| Time | PRICE | SIZE | SIDE | PROFIT | |
|---|---|---|---|---|---|
| 0 | 2003-12-01 00:00:00.000 | 13.5 | 200 | B | -2700 |
| 1 | 2003-12-01 00:00:00.001 | 13.6 | 100 | S | -1340 |
| 2 | 2003-12-01 00:00:00.002 | 13.3 | 150 | B | -3335 |
| 3 | 2003-12-01 00:00:00.003 | 14.0 | 200 | S | -535 |
The variable ‘PROFIT’ keeps a running total. In other words, it aggregates state across trades.
Note that the same can be accomplished without variables by keeping the running total in a separate column.
trd = otp.Ticks({'PRICE': [13.5, 13.6, 13.3, 14.0],
'SIZE': [200, 100, 150, 200],
'SIDE': ['B', 'S', 'B', 'S']})
trd['VALUE'] = trd.apply(
lambda t: t['PRICE'] * t['SIZE'] if t['SIDE'] == 'S' else -(t['PRICE'] * t['SIZE'])
)
trd = trd.agg({'PROFIT': otp.agg.sum('VALUE')}, running=True)
otp.run(trd)
| Time | PROFIT | |
|---|---|---|
| 0 | 2003-12-01 00:00:00.000 | -2700.0 |
| 1 | 2003-12-01 00:00:00.001 | -1340.0 |
| 2 | 2003-12-01 00:00:00.002 | -3335.0 |
| 3 | 2003-12-01 00:00:00.003 | -535.0 |
Below is an example of using variables that cannot be as easily implemented with running totals: remembering the value of the last tick and referring to it after grouping/aggregation.
q = otp.Ticks(X=[-1, 3, -3, 4, 2], Y=[0, 1, 1, 0, 3])
q.state_vars['last_x'] = 0
q.state_vars['last_x'] = q['X']
q = q.high('X', group_by=['Y'])
q['last_x'] = q.state_vars['last_x']
otp.run(q)
| Time | X | Y | last_x | |
|---|---|---|---|---|
| 0 | 2003-12-01 00:00:00.001 | 3 | 1 | 2 |
| 1 | 2003-12-01 00:00:00.003 | 4 | 0 | 2 |
| 2 | 2003-12-01 00:00:00.004 | 2 | 3 | 2 |
Dictionaries / Maps (aka tick sets)#
Looking up static data for every tick#
A map can be created with keys taken from one or more columns and holding entire ticks as values.
The example below uses exchange reference data to create a map keyed by exchange code.
exchanges = otp.Ticks(EXCHANGE=['A', 'B', 'C', 'D', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'P', 'Q', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'],
NAME=['NYSE American (Amex)', 'Nasdaq BX', 'NYSE National (NSX)', 'FINRA ADF + NYSE/Nasdaq TRFs', 'MIAX Pearl', 'International Securities Exchange', 'Cboe EDGA', 'Cboe EDGX', 'Long-Term Stock Exchange (LTSE)', 'NYSE Chicago', 'New York Stock Exchange', 'NYSE Arca', 'Nasdaq (Tape C securities)', 'Consolidated Tape System (CTS)', 'Nasdaq (Tape A,B securities)', 'Members Exchange (MEMX)', "The Investors' Exchange (IEX)", 'CBOE Stock Exchange (CBSX)', 'Nasdaq PSX', 'Cboe BYX', 'Cboe BZX'])
exchanges['LOCATION'] = 'US'
otp.run(exchanges)
| Time | EXCHANGE | NAME | LOCATION | |
|---|---|---|---|---|
| 0 | 2003-12-01 00:00:00.000 | A | NYSE American (Amex) | US |
| 1 | 2003-12-01 00:00:00.001 | B | Nasdaq BX | US |
| 2 | 2003-12-01 00:00:00.002 | C | NYSE National (NSX) | US |
| 3 | 2003-12-01 00:00:00.003 | D | FINRA ADF + NYSE/Nasdaq TRFs | US |
| 4 | 2003-12-01 00:00:00.004 | H | MIAX Pearl | US |
| 5 | 2003-12-01 00:00:00.005 | I | International Securities Exchange | US |
| 6 | 2003-12-01 00:00:00.006 | J | Cboe EDGA | US |
| 7 | 2003-12-01 00:00:00.007 | K | Cboe EDGX | US |
| 8 | 2003-12-01 00:00:00.008 | L | Long-Term Stock Exchange (LTSE) | US |
| 9 | 2003-12-01 00:00:00.009 | M | NYSE Chicago | US |
| 10 | 2003-12-01 00:00:00.010 | N | New York Stock Exchange | US |
| 11 | 2003-12-01 00:00:00.011 | P | NYSE Arca | US |
| 12 | 2003-12-01 00:00:00.012 | Q | Nasdaq (Tape C securities) | US |
| 13 | 2003-12-01 00:00:00.013 | S | Consolidated Tape System (CTS) | US |
| 14 | 2003-12-01 00:00:00.014 | T | Nasdaq (Tape A,B securities) | US |
| 15 | 2003-12-01 00:00:00.015 | U | Members Exchange (MEMX) | US |
| 16 | 2003-12-01 00:00:00.016 | V | The Investors' Exchange (IEX) | US |
| 17 | 2003-12-01 00:00:00.017 | W | CBOE Stock Exchange (CBSX) | US |
| 18 | 2003-12-01 00:00:00.018 | X | Nasdaq PSX | US |
| 19 | 2003-12-01 00:00:00.019 | Y | Cboe BYX | US |
| 20 | 2003-12-01 00:00:00.020 | Z | Cboe BZX | US |
We will add exchange name to trades. First let’s examine the trades.
q = otp.DataSource('NYSE_TAQ', tick_type='TRD')
q = q[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
otp.run(q, start=s, end=e, symbols=['SPY'])
| Time | PRICE | SIZE | COND | EXCHANGE | |
|---|---|---|---|---|---|
| 0 | 2023-05-15 09:30:00.000178688 | 412.22 | 100 | T | P |
| 1 | 2023-05-15 09:30:00.000776704 | 412.22 | 247 | Z | |
| 2 | 2023-05-15 09:30:00.003603456 | 412.22 | 100 | T | T |
| 3 | 2023-05-15 09:30:00.006352128 | 412.24 | 1 | I | K |
| 4 | 2023-05-15 09:30:00.007128064 | 412.24 | 3 | I | K |
| ... | ... | ... | ... | ... | ... |
| 310 | 2023-05-15 09:30:00.934032640 | 412.27 | 160 | T | T |
| 311 | 2023-05-15 09:30:00.975609344 | 412.24 | 2 | I | D |
| 312 | 2023-05-15 09:30:00.980264448 | 412.27 | 1 | I | D |
| 313 | 2023-05-15 09:30:00.985391616 | 412.28 | 100 | T | |
| 314 | 2023-05-15 09:30:00.985394944 | 412.28 | 100 | Q | T |
315 rows × 5 columns
The value of the EXCHANGE field from trades will be used to look up the name of the corresponding exchange.
q = otp.DataSource('NYSE_TAQ', tick_type='TRD')
q = q[['PRICE','SIZE','COND','EXCHANGE']]
exchanges = otp.Ticks(EXCHANGE=['A', 'B', 'C', 'D', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'P', 'Q', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'],
NAME=['NYSE American (Amex)', 'Nasdaq BX', 'NYSE National (NSX)', 'FINRA ADF + NYSE/Nasdaq TRFs', 'MIAX Pearl', 'International Securities Exchange', 'Cboe EDGA', 'Cboe EDGX', 'Long-Term Stock Exchange (LTSE)', 'NYSE Chicago', 'New York Stock Exchange', 'NYSE Arca', 'Nasdaq (Tape C securities)', 'Consolidated Tape System (CTS)', 'Nasdaq (Tape A,B securities)', 'Members Exchange (MEMX)', "The Investors' Exchange (IEX)", 'CBOE Stock Exchange (CBSX)', 'Nasdaq PSX', 'Cboe BYX', 'Cboe BZX'])
q.state_vars['exchanges'] = otp.state.tick_set('latest', 'EXCHANGE', otp.eval(exchanges))
q['exchange_name'] = q.state_vars['exchanges'].find('NAME', 'unknown')
otp.run(q, start=s, end=e, symbols=['SPY'])
| Time | PRICE | SIZE | COND | EXCHANGE | exchange_name | |
|---|---|---|---|---|---|---|
| 0 | 2023-05-15 09:30:00.000178688 | 412.22 | 100 | T | P | NYSE Arca |
| 1 | 2023-05-15 09:30:00.000776704 | 412.22 | 247 | Z | Cboe BZX | |
| 2 | 2023-05-15 09:30:00.003603456 | 412.22 | 100 | T | T | Nasdaq (Tape A,B securities) |
| 3 | 2023-05-15 09:30:00.006352128 | 412.24 | 1 | I | K | Cboe EDGX |
| 4 | 2023-05-15 09:30:00.007128064 | 412.24 | 3 | I | K | Cboe EDGX |
| ... | ... | ... | ... | ... | ... | ... |
| 310 | 2023-05-15 09:30:00.934032640 | 412.27 | 160 | T | T | Nasdaq (Tape A,B securities) |
| 311 | 2023-05-15 09:30:00.975609344 | 412.24 | 2 | I | D | FINRA ADF + NYSE/Nasdaq TRFs |
| 312 | 2023-05-15 09:30:00.980264448 | 412.27 | 1 | I | D | FINRA ADF + NYSE/Nasdaq TRFs |
| 313 | 2023-05-15 09:30:00.985391616 | 412.28 | 100 | T | Nasdaq (Tape A,B securities) | |
| 314 | 2023-05-15 09:30:00.985394944 | 412.28 | 100 | Q | T | Nasdaq (Tape A,B securities) |
315 rows × 6 columns
Checking if ticks are present in another time series#
A tick set can be used to check if (a small number of) ticks are present in another time series with potentially different time stamps.
We generate two “special” ticks.
ticks = otp.Ticks(PRICE=[412.22, 400], SIZE=[247, 1000], INDEX=[1, 2])
otp.run(ticks)
| Time | PRICE | SIZE | INDEX | |
|---|---|---|---|---|
| 0 | 2003-12-01 00:00:00.000 | 412.22 | 247 | 1 |
| 1 | 2003-12-01 00:00:00.001 | 400.00 | 1000 | 2 |
We then check if any of the trades match these ticks by PRICE and SIZE.
q = otp.DataSource('NYSE_TAQ', tick_type='TRD')
q = q[['PRICE', 'SIZE', 'COND', 'EXCHANGE']]
q.state_vars['special_ticks'] = otp.state.tick_set('latest', ['PRICE', 'SIZE'], otp.eval(ticks))
q['special'] = q.state_vars['special_ticks'].find('INDEX', -1)
q, _ = q[q['special'] != -1]
otp.run(q, start=s, end=e, symbols=['SPY'])
| Time | PRICE | SIZE | COND | EXCHANGE | special | |
|---|---|---|---|---|---|---|
| 0 | 2023-05-15 09:30:00.000776704 | 412.22 | 247 | Z | 1 |
Lists and Queues#
Lists and double-ended queues (deques) can be used to keep track of collections ticks.