# Variables and Data Structures ## Variables (aka 'state variables') Variables can be used to keep track of state across ticks. The example below shows how we may keep track of P&L. ```ipython3 import onetick.py as otp s = otp.dt(2024, 2, 1, 9, 30) e = otp.dt(2024, 2, 1, 9, 30, 1) trd = otp.Ticks({'PRICE': [13.5, 13.6, 13.3, 14.0], 'SIZE': [200, 100, 150, 200], 'SIDE': ['B', 'S', 'B', 'S']}) trd.state_vars['PROFIT'] = 0 trd.state_vars['PROFIT'] += trd.apply( lambda t: t['PRICE'] * t['SIZE'] if t['SIDE'] == 'S' else -(t['PRICE'] * t['SIZE']) ) trd['PROFIT'] = trd.state_vars['PROFIT'] otp.run(trd) ``` ```myst-ansi Time PRICE SIZE SIDE PROFIT 0 2003-12-01 00:00:00.000 13.5 200 B -2700 1 2003-12-01 00:00:00.001 13.6 100 S -1340 2 2003-12-01 00:00:00.002 13.3 150 B -3335 3 2003-12-01 00:00:00.003 14.0 200 S -535 ``` The variable 'PROFIT' keeps a running total. In other words, it aggregates state across trades. Note that the same can be accomplished without variables by keeping the running total in a separate column. ```ipython3 trd = otp.Ticks({'PRICE': [13.5, 13.6, 13.3, 14.0], 'SIZE': [200, 100, 150, 200], 'SIDE': ['B', 'S', 'B', 'S']}) trd['VALUE'] = trd.apply( lambda t: t['PRICE'] * t['SIZE'] if t['SIDE'] == 'S' else -(t['PRICE'] * t['SIZE']) ) trd = trd.agg({'PROFIT': otp.agg.sum('VALUE')}, running=True) otp.run(trd) ``` ```myst-ansi Time PROFIT 0 2003-12-01 00:00:00.000 -2700.0 1 2003-12-01 00:00:00.001 -1340.0 2 2003-12-01 00:00:00.002 -3335.0 3 2003-12-01 00:00:00.003 -535.0 ``` Below is an example of using variables that cannot be as easily implemented with running totals: remembering the value of the last tick and referring to it after grouping/aggregation. ```ipython3 q = otp.Ticks(X=[-1, 3, -3, 4, 2], Y=[0, 1, 1, 0, 3]) q.state_vars['last_x'] = 0 q.state_vars['last_x'] = q['X'] q = q.high('X', group_by=['Y']) q['last_x'] = q.state_vars['last_x'] otp.run(q) ``` ```myst-ansi Time X Y last_x 0 2003-12-01 00:00:00.001 3 1 2 1 2003-12-01 00:00:00.003 4 0 2 2 2003-12-01 00:00:00.004 2 3 2 ``` # Dictionaries / Maps (aka `tick sets`) ## Looking up static data for every tick A map can be created with keys taken from one or more columns and holding entire ticks as values. The example below uses exchange reference data to create a map keyed by exchange code. ```ipython3 exchanges = otp.Ticks( EXCHANGE=['A', 'B', 'C', 'D', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'P', 'Q', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'], NAME=['NYSE American (Amex)', 'Nasdaq BX', 'NYSE National (NSX)', 'FINRA ADF + NYSE/Nasdaq TRFs', 'MIAX Pearl', 'International Securities Exchange', 'Cboe EDGA', 'Cboe EDGX', 'Long-Term Stock Exchange (LTSE)', 'NYSE Chicago', 'New York Stock Exchange', 'NYSE Arca', 'Nasdaq (Tape C securities)', 'Consolidated Tape System (CTS)', 'Nasdaq (Tape A,B securities)', 'Members Exchange (MEMX)', "The Investors' Exchange (IEX)", 'CBOE Stock Exchange (CBSX)', 'Nasdaq PSX', 'Cboe BYX', 'Cboe BZX'] ) exchanges['LOCATION'] = 'US' otp.run(exchanges) ``` ```myst-ansi Time EXCHANGE NAME LOCATION 0 2003-12-01 00:00:00.000 A NYSE American (Amex) US 1 2003-12-01 00:00:00.001 B Nasdaq BX US 2 2003-12-01 00:00:00.002 C NYSE National (NSX) US 3 2003-12-01 00:00:00.003 D FINRA ADF + NYSE/Nasdaq TRFs US 4 2003-12-01 00:00:00.004 H MIAX Pearl US 5 2003-12-01 00:00:00.005 I International Securities Exchange US 6 2003-12-01 00:00:00.006 J Cboe EDGA US 7 2003-12-01 00:00:00.007 K Cboe EDGX US 8 2003-12-01 00:00:00.008 L Long-Term Stock Exchange (LTSE) US 9 2003-12-01 00:00:00.009 M NYSE Chicago US 10 2003-12-01 00:00:00.010 N New York Stock Exchange US 11 2003-12-01 00:00:00.011 P NYSE Arca US 12 2003-12-01 00:00:00.012 Q Nasdaq (Tape C securities) US 13 2003-12-01 00:00:00.013 S Consolidated Tape System (CTS) US 14 2003-12-01 00:00:00.014 T Nasdaq (Tape A,B securities) US 15 2003-12-01 00:00:00.015 U Members Exchange (MEMX) US 16 2003-12-01 00:00:00.016 V The Investors' Exchange (IEX) US 17 2003-12-01 00:00:00.017 W CBOE Stock Exchange (CBSX) US 18 2003-12-01 00:00:00.018 X Nasdaq PSX US 19 2003-12-01 00:00:00.019 Y Cboe BYX US 20 2003-12-01 00:00:00.020 Z Cboe BZX US ``` We will add exchange name to trades. First let's examine the trades. ```ipython3 q = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD') q = q[['PRICE', 'SIZE', 'COND', 'EXCHANGE']] otp.run(q, start=s, end=e, symbols=['AAPL']) ``` ```myst-ansi Time PRICE SIZE COND EXCHANGE 0 2024-02-01 09:30:00.000961260 184.010 302 @FT P 1 2024-02-01 09:30:00.000961491 184.000 100 @FT P 2 2024-02-01 09:30:00.000961701 184.000 1 @FTI P 3 2024-02-01 09:30:00.000973163 184.000 1 @FTI P 4 2024-02-01 09:30:00.000973355 184.000 5 @FTI P .. ... ... ... ... ... 574 2024-02-01 09:30:00.987184691 183.900 9 @F I K 575 2024-02-01 09:30:00.990378350 183.920 1 @ I D 576 2024-02-01 09:30:00.991941892 183.935 1 @ I D 577 2024-02-01 09:30:00.993785116 183.905 300 @ D 578 2024-02-01 09:30:00.996512511 183.934 5 @ I D [579 rows x 5 columns] ``` The value of the `EXCHANGE` field from trades will be used to look up the name of the corresponding exchange. ```ipython3 q = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD') q = q[['PRICE','SIZE','COND','EXCHANGE']] exchanges = otp.Ticks( EXCHANGE=['A', 'B', 'C', 'D', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'P', 'Q', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'], NAME=['NYSE American (Amex)', 'Nasdaq BX', 'NYSE National (NSX)', 'FINRA ADF + NYSE/Nasdaq TRFs', 'MIAX Pearl', 'International Securities Exchange', 'Cboe EDGA', 'Cboe EDGX', 'Long-Term Stock Exchange (LTSE)', 'NYSE Chicago', 'New York Stock Exchange', 'NYSE Arca', 'Nasdaq (Tape C securities)', 'Consolidated Tape System (CTS)', 'Nasdaq (Tape A,B securities)', 'Members Exchange (MEMX)', "The Investors' Exchange (IEX)", 'CBOE Stock Exchange (CBSX)', 'Nasdaq PSX', 'Cboe BYX', 'Cboe BZX'] ) q.state_vars['exchanges'] = otp.state.tick_set('latest', 'EXCHANGE', otp.eval(exchanges)) q['exchange_name'] = q.state_vars['exchanges'].find('NAME', 'unknown') otp.run(q, start=s, end=e, symbols=['AAPL']) ``` ```myst-ansi Time PRICE SIZE COND EXCHANGE exchange_name 0 2024-02-01 09:30:00.000961260 184.010 302 @FT P NYSE Arca 1 2024-02-01 09:30:00.000961491 184.000 100 @FT P NYSE Arca 2 2024-02-01 09:30:00.000961701 184.000 1 @FTI P NYSE Arca 3 2024-02-01 09:30:00.000973163 184.000 1 @FTI P NYSE Arca 4 2024-02-01 09:30:00.000973355 184.000 5 @FTI P NYSE Arca .. ... ... ... ... ... ... 574 2024-02-01 09:30:00.987184691 183.900 9 @F I K Cboe EDGX 575 2024-02-01 09:30:00.990378350 183.920 1 @ I D FINRA ADF + NYSE/Nasdaq TRFs 576 2024-02-01 09:30:00.991941892 183.935 1 @ I D FINRA ADF + NYSE/Nasdaq TRFs 577 2024-02-01 09:30:00.993785116 183.905 300 @ D FINRA ADF + NYSE/Nasdaq TRFs 578 2024-02-01 09:30:00.996512511 183.934 5 @ I D FINRA ADF + NYSE/Nasdaq TRFs [579 rows x 6 columns] ``` ## Checking if ticks are present in another time series A tick set can be used to check if (a small number of) ticks are present in another time series with potentially different time stamps. We generate two "special" ticks. ```ipython3 ticks = otp.Ticks(PRICE=[412.22, 400], SIZE=[247, 1000], INDEX=[1, 2]) otp.run(ticks) ``` ```myst-ansi Time PRICE SIZE INDEX 0 2003-12-01 00:00:00.000 412.22 247 1 1 2003-12-01 00:00:00.001 400.00 1000 2 ``` We then check if any of the trades match these ticks by `PRICE` and `SIZE`. ```ipython3 q = otp.DataSource('US_COMP_SAMPLE', tick_type='TRD') q = q[['PRICE', 'SIZE', 'COND', 'EXCHANGE']] q.state_vars['special_ticks'] = otp.state.tick_set('latest', ['PRICE', 'SIZE'], otp.eval(ticks)) q['special'] = q.state_vars['special_ticks'].find('INDEX', -1) q = q.where(q['special'] != -1) otp.run(q, start=s, end=e, symbols=['AAPL']) ``` ```myst-ansi Empty DataFrame Columns: [PRICE, SIZE, COND, EXCHANGE, special, Time] Index: [] ``` # Lists and Queues Lists and double-ended queues (deques) can be used to keep track of collections ticks. ## Lists and Queues Use Cases [Realized P&L (FIFO)](use_cases/realized_profit.md)