Publish/Subscribe -- collect data and distribute to subscribers.

Tickerplant does not store data. The real-time database does that.

Tickerplant can have direct subscribers or users can query the real time 
database.

Tickerplant also stores data to a log file but purges in-memory tables.

Real-time database on start-up requests data from tickerplant by
reading it from the log file.

Real-time subscriber can ask for all data or just some of it.
Subscription is on a table and list of symbols.

Tickerplant communicates with subscribers on a timer loop so subscribers
may encounter a delay before they get the data.

Chained tickerplant may be used if there can be ad hoc queries.


In the product,
when the real time database (RTD) first talks, it makes a synchronous
call and asks for the schemas of all the tables in
the tickerplant.
RTD gets the location of the log and the number of messages
logged so far.
So the tickerplant client must define an upd function.

It keeps receiving until .u.end which means all the data has been given.
Update messages are of the form:
(`upd;`table_name;table_data)
Typically the client will save to disk, transfer data to hdb and perhaps discard
its tables.


Understand a slightly simplified version of this  as follows.


To run this, have four terminal windows open
and run

window 1 tickerplant at port 5010:
q ticker.q -p 5010 

window 2 client wants a real time database:
q cx.q rdb 

window 3 client wants high low close volume:
q cx.q hlcv / client wants high low close volume

window 4 feeder:
q feed.q

For example, if we specify hlcv, then t becomes trade s is `all meaning
all symbols (but that could be changed).

q feed.q / generates data and sends to ticker

feed produces data and then stops.
So, to show what is going on, shut down feed.q and then start it 
again.
If you don't want so much time to go to the screen, take away the show hlcv
in the "hlcv" client.


How It All Works

Subscription

Each subscription request
from a client (running cx.q) specifies a table, a list of symbols,
and information about where to send the requested information
(interprocess communication information about the client).

Here are the details.
That subscription request happens at cx.q at this line:
{h("sub";x;s)} each t;
where t and s are specified elsewhere in cx.q
The two explicit arguments here are a table (replaced by t) 
and a set of symbols.
That calls the routine sub in ticker.q (with implicit arguments x, y, z):
Sub:`quote`trade!()
sub:{Sub[x],:enlist x,y,neg .z.w}
So for each table t (which is first argument (x) of sub), enlist
to Sub[t] the list (t, symbols of t, return address of client process).

So, the information flow for subscription is client to tickerplant
-- cx.q to ticker.q

New Data

Later, feed.q will send an update to ticker.q as follows:
h:neg hopen `::5010
establishes an async connection at port 5010 on this machine.
(Machine name is between the two colons.)

feed.q will in fact send the data in the function feed (which should be
changed when you're sending real data).
In the file the function feed performs a conditional statement
(based on whether rand 2 returns a 0 or a 1) and invokes upd on
the client either
to trade or to quote. 
What it sends is dictated by the functions t or q, but in real life
this would come from actual data.

When an update arrives (from the feeder) at the tickerplant, 
it arrives as a call to the function upd.
The first argument of upd is the table to be updated.
The second is the row information.

The dataflow so far is feed.q --> tick.q 

Chained Tickerplant

The way this works is that the tickerchainmaster.q knows about
one or more subsidiary ticker.q
processes and sends to them exactly what the feeder sends.
The subsidiary ticker.q are classic ones.

So the flow is
feed.q --> tickerchainmaster.q --> one or more ticker.q --> cxchained.q

Clients can also subscribe directly to tickerchainmaster.q
cxchained.q and cx.q differ only in the port they go to.
cx.q goes to port 5010 which is where tickerchainmaster.q runs.
cxchained.q goes to port 5012 where the current single instance
of ticker.q runs.

window 1 chained tickerplant: 
q ticker.q -p 5012 

window 2 master tickerplant that receives from feed: 
q tickerchainmaster.q -p 5010


window 3:
/ client wants high low close volume from ticker
/ works off of chain master because of port 5010
q cx.q hlcv 

window 4:
/ client wants high low close volume from ticker
/ this differs from cx.q only in that it requests from port 5012
q cxchained.q hlcv 

window 5:
q feed.q


Challenge problem suggested by Jeff Shmain: Suppose that one tickerplant dies.
Could there be a way to have a client subscribe to two tickerplants
and distinguish among them using sequence numbers or some other 
such mechanism?
Thus, the basic design could go something like this:
a single client sends subscriptions to two tickerplants.
Each one sends updates. The client update function (upd) checks whether
an input row has already been received before inserting it.
If the records have keys, you can use the "upsert" instruction.

ALTERNATIVE DESIGN (read on your own)

The chained tickerplant subscribes to the ticker.q (so executes after
ticker.q) then accepts subscriptions and passes them on.
So it's partly a client and partly a tickerplant.

window 1:
q ticker.q -p 5010 / tickerplant at port 5010

window2:
q tickerchained.q -p 5012

window 3:
q cx.q hlcv / client wants high low close volume from ticker

window 4:
q cxchained.q hlcv / client wants high low close volume from ticker
	/ this differs from cx.q only in that it requests from port 5012

window 5:
q feed.q


Information note: In the kdb+/tick package,

tick.k -- tickerplant itself
r.k -- real time database
u.k -- contains the definitions for the publisher/subscriber functions.
ssl.q -- receives raw data from the feed

tables:
taq: trade and quote data
sym: simplified trad and quote data
fx: foreign exchange data
lvl2: level 2 data


