10 minute rollup

calculate the last price and sum of size of 10 minute intervals of trade
select last price, sum size by 10 xbar time.minute from trade where sym=`MSFT

hourly mean variance of some symbol
select mean:avg price, variance:var price, stdDev:dev price by date, hour:time.hh from trade where sym=`AIG

Average total trade volume over a certain week, for each 30 minute bucket for MSFT
(select sum size by 30 xbar time.minute from trade where date within 2006.08.07 2006.08.11,sym=`MSFT)% sum date within 2006.08.07 2006.08.11

Select the two heaviest traded stocks
select sym from 2# desc select sum size by sym from trade

This dyadic function shifts a vector by n elements, for example
select price-5 xprev price from trade where sym=`IBM

Generate a NBBO table for a certain month: (National best bid and offer)
nbbo: select ask: min ask,bid: max bid by date,sym,time from quote where date.month=2006.08m,asize>0,bsize>0

non-first normal form can be faster
q) \t do[10000;r1:select avg price by sym from trade where sym in `AIG`IBM] 1862
compared to
q) \t nestedtrade:select price,size by sym from trade 0 q) \t do[10000;r2:select sym,each[avg] price from nestedtrade where sym in `AIG`IBM] 180

=======

Set up a console

q trade.q -p 5001

http://localhost:5001
http://localhost:5001/?select from trade where sym=`IBM

export to excel:

http://localhost:5001/.csv?select from trade where sym=`IBM

\a set of tables
\c 20 200 how much to display
\v all global variable
\date -- operating system date ommand

========

Interlanguage access -- do excel, web, and maybe java
p. 112 of fdkdbtutorial

// build (time;sym;price;size) record
Object[]x={new Time(System.currentTimeMillis()%86400000),"xx",new
Double(93.5),new Integer(300)};
c.k("insert","trade",x);	// insert[`trade;x]
clients can also read incoming messages:
r=c.k();

for example, kdb+tick clients do:
String[]syms={"IBM","MSFT",..};
c.k(".u.sub","trade",syms);	// subscribe
while(1){Object r=c.k();..	// process incoming


p. 126 of the tutorial

import c; // That is the critical java to kdb module to import

// I don't know which of these to import
import java.awt.*; 
import java.awt.event.ActionEvent; 
import java.io.IOException;
import javax.swing.*; 
import javax.swing.table.DefaultTableModel;
// end of I don't know what

String host = "localhost";
int port = 6001;
qConn = new c(host, port); connects to a database specified by host and port

Object o = qConn.k(sSql); 
	// access the database and gets info based on sql query

c.Flip d = c.td(o); // columns are d.x and rows are d.y

int numcols = c.n(d.x); 
int numrows = c.n(d.y[0]);

Object[][] myobj = new Object[numrows][numcols];

for (int rowid = 0; rowid < numrows; rowid++) {
  for (int colid = 0; colid < numcols; colid++) {
	myobj[rowid][colid] = c.at(d.y[colid], rowid); 
		// fishes out the colid field of row rowid
  };
};



clients (see http://www.studio4kdb.com/ExcelRTDServer.htm)

Probably don't need this:

From chris@kx.com  Sun Mar 17 06:25:28 2013
Received: from mx2.cims.nyu.edu (mx2.cims.nyu.edu [128.122.49.96])
	by mail.cims.nyu.edu (8.14.5+Sun/8.14.5) with ESMTP id r2HAPSlQ004538
	for <shasha@mail.cims.nyu.edu>; Sun, 17 Mar 2013 06:25:28 -0400 (EDT)
Received: from mail-ee0-f51.google.com (mail-ee0-f51.google.com [74.125.83.51])
	by mx2.cims.nyu.edu (8.14.4+Sun/8.14.4) with ESMTP id r2HAPNTI027057
	for <shasha@cs.nyu.edu>; Sun, 17 Mar 2013 06:25:23 -0400 (EDT)
Received: by mail-ee0-f51.google.com with SMTP id d17so2140481eek.10
        for <shasha@cs.nyu.edu>; Sun, 17 Mar 2013 03:25:17 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=google.com; s=20120113;
        h=x-received:mime-version:in-reply-to:references:from:date:message-id
         :subject:to:cc:content-type:x-gm-message-state;
        bh=XIV8/avRzAXMDYktTDqmGdYvGantyKTQrh4uKoJUvlY=;
        b=C6frA69d7/jFscwbGhicgp//j/hC9G/BsSN0L73WQNGhFOOUf+FP1emSjF4gbdl2OR
         +sID7m7lLk16CaPAo5VEqR2pHEA3Eu9h1a5tc/M5JHqpFeO5KJONGg5N4bkmJte8QdUy
         nZzB8AFKEXXuNIz9UKteuOFUf946Kl3eqH4Cw1MLroCZLSIQTmHpM7QmqvLWbrlKk6LV
         SNq1Lgg64DKtl5S3z6X0uYBdWRgwpPid6Y9fUY12D4gsbSl1bsBOeSEKEGP9dNL7AX9x
         mgIzlH4QjrZfPbzQtJ8RdnV77fggFnoNnphdFnyCXEgrnu24H6hKBEa8KHJy+hppC+Sm
         HkYg==
X-Received: by 10.14.223.69 with SMTP id u45mr36907445eep.23.1363515917688;
 Sun, 17 Mar 2013 03:25:17 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.188.7 with HTTP; Sun, 17 Mar 2013 03:24:37 -0700 (PDT)
In-Reply-To: <201303170013.r2H0DMFE017453@crunchy12.cims.nyu.edu>
References: <201303170013.r2H0DMFE017453@crunchy12.cims.nyu.edu>
From: Chris Burke <chris@kx.com>
Date: Sun, 17 Mar 2013 18:24:37 +0800
Message-ID: <CAOAyECOcj5e=+ngiGrse=AOoXPjXUhQxpA-t8aeks-OW=egNLg@mail.gmail.com>
Subject: Re: r to q
To: Dennis Shasha <shasha@courant.nyu.edu>
Cc: shasha@cs.nyu.edu
Content-Type: multipart/alternative; boundary=047d7b604fbaeb861004d81c49e8
X-Gm-Message-State: ALoCoQmq+WBUs41d3dNmc7PHlPm4Z3KZ9VwW0fa+FoSbg3ryRQyvhhodo3Wvrz3AypS1SeFZmha6
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.1 (mx2.cims.nyu.edu [128.122.49.96]); Sun, 17 Mar 2013 06:25:23 -0400 (EDT)
X-Virus-Scanned: clamav-milter 0.97.4 at mx2
X-Virus-Status: Clean
X-Scanned-By: MIMEDefang 2.73 on 128.122.49.96
Status: R
Content-Length: 9646

--047d7b604fbaeb861004d81c49e8
Content-Type: text/plain; charset=UTF-8

Dennis

What is happening is that R is built with gcc, and R shared libraries must
also be built with gcc. The problem you see is a bug in older gcc libraries
that has now been fixed. Unfortunately, the version of gcc that comes with
OSX is out of date and still has the bug.

Thus the problem is with gcc, and is not specific to your version of R or
OSX.

You should be able to fix the problem by simply installing a recent release
of gcc. You can do this via Macports. It should not be necessary to
recompile qserver.so, though it should be trivial to do so. Could you try
this experiment?

I agree this is unsatisfactory for a classroom where people bring their own
Macs. However, you could still use R for demos, and could advise anyone who
wanted to use the R interface to upgrade their gcc installation.

Regards

Chris


On Sun, Mar 17, 2013 at 8:13 AM, Dennis Shasha <shasha@courant.nyu.edu>wrote:

>
>
> Chris,
> OSX 10.6.8. My worry is not so much for me as that I can't teach students
> to use software if they are not at some particular version of OS and
> compiler.
> Thanks,
> Dennis
> From chris@kx.com  Sat Mar 16 18:07:15 2013
> X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
>         d=google.com; s=20120113;
>
> h=x-received:mime-version:in-reply-to:references:from:date:message-id
>          :subject:to:cc:content-type:x-gm-message-state;
>         bh=71wSAgAcVTdzFTENy6y2sp2imBZvf+9jhQW0rp71jPY=;
>
> b=eGl75j6NLKXrsc215ofg+M/iu7l7zNZg7RZalH1HRtKmigN9LkBj95q9wMEMwGAOZy
>
>  kFX1bJpxpN0caUzV2doTW0kGqFOT06nRzlayQ2NQ+CQCgdVltydUtk1wGo4fsvuJXexs
>
>  El6W8344c06LDlUHWkOgsq4xRWR/S9zDbppc5X5v3lq5KrWkren32m54tzPc3gqG8LXz
>
>  v5iVcRNUELctkvKEz50DKeChFWJF5pTpD11XZ1snbUvXOM/mpp6EKWugEfbQPYbnse6x
>
>  3dVc4tdO8/Yf6gRoYCV9tORXH30UeAeGWbgxxw2T/uyg75H48pnaFth1KHkj6XSOhQG4
>          /Nlg==
> X-Received: by 10.14.215.193 with SMTP id
> e41mr32601170eep.32.1363471627613;
>  Sat, 16 Mar 2013 15:07:07 -0700 (PDT)
> MIME-Version: 1.0
> From: Chris Burke <chris@kx.com>
> Date: Sun, 17 Mar 2013 06:06:27 +0800
> Subject: Re: r to q
> To: Dennis Shasha <shasha@courant.nyu.edu>
> Cc: shasha@cs.nyu.edu
> X-Gm-Message-State:
> ALoCoQle4s4kBZMfiAAg4Cxl1RjmFTYy1nVdmxFq3M+v6xZiA32vU2blVFTwKr42ZA/VG/d87sNG
> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.1 (
> mx.cims.nyu.edu [128.122.49.99]); Sat, 16 Mar 2013 18:07:13 -0400 (EDT)
> X-Virus-Scanned: clamav-milter 0.97.4 at mx
> X-Virus-Status: Clean
> X-Scanned-By: MIMEDefang 2.73 on 128.122.49.99
>
> --e89a8f647ac306b6a104d811fab5
> Content-Type: text/plain; charset=UTF-8
>
> Hi Dennis
>
> What version of OSX do you have?
>
> Chris
>
> On Sun, Mar 17, 2013 at 12:26 AM, Dennis Shasha <shasha@courant.nyu.edu
> >wrote:
>
> >
> > Dear Chris,
> > In your mail you said:
> >
> > =========
> > The attached was built with gcc 4.7 for R 2.15.3 on osx 10.8.2. This
> allows
> > R to connect to a q server from an R client, see examples in the zip.
> > ==========
> >
> > Naively, I thought that I could use a slightly older mac os
> > and R package without any special compilation effort.
> > So, I tried the following
> >
> > q sp.q -p 5000
> >
> > R
> > -- and then I tried dyn.load("qserver.so")
> >
> > but I get the following error:
> >
> > Error in dyn.load("qserver.so") :
> >   unable to load shared object
> > '/Users/dennisshasha/q/smartiesinterconnect.d/qserver.so':
> >   dlopen(/Users/dennisshasha/q/smartiesinterconnect.d/qserver.so, 6):
> > Symbol not found: ___emutls_get_address
> >   Referenced from:
> /Users/dennisshasha/q/smartiesinterconnect.d/qserver.so
> >   Expected in: /usr/lib/libSystem.B.dylib
> >  in /Users/dennisshasha/q/smartiesinterconnect.d/qserver.so
> >
> >
> > So, the question is:
> > what is the overall procedure? For example, should I put qserver.so
> > somewhere
> > else?
> >
> > Sorry for the elementary question.
> > I've never connected any language to q.
> >
> > Thanks,
> > Dennis
>

--047d7b604fbaeb861004d81c49e8
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div>Dennis</div><div><br></div><div>What is happening is that R is built w=
ith gcc, and R shared libraries must also be built with gcc. The problem yo=
u see is a bug in older gcc libraries that has now been fixed. Unfortunatel=
y, the version of gcc that comes with OSX is out of date and still has the =
bug.</div>

<div><br></div><div>Thus the problem is with gcc, and is not specific to yo=
ur version of R or OSX.=C2=A0</div><div><br></div><div>You should be able t=
o fix the problem by simply installing a recent release of gcc. You can do =
this via Macports. It should not be necessary to recompile qserver.so, thou=
gh it should be trivial to do so. Could you try this experiment?</div>

<div><br></div><div>I agree this is unsatisfactory for a classroom where pe=
ople bring their own Macs. However, you could still use R for demos, and co=
uld advise anyone who wanted to use the R interface to upgrade their gcc in=
stallation.</div>

<div><br></div><div>Regards</div><div><br></div><div>Chris</div><div><br></=
div><br><div class=3D"gmail_quote">On Sun, Mar 17, 2013 at 8:13 AM, Dennis =
Shasha <span dir=3D"ltr">&lt;<a href=3D"mailto:shasha@courant.nyu.edu" targ=
et=3D"_blank">shasha@courant.nyu.edu</a>&gt;</span> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><br>
<br>
Chris,<br>
OSX 10.6.8. My worry is not so much for me as that I can&#39;t teach studen=
ts<br>
to use software if they are not at some particular version of OS and compil=
er.<br>
Thanks,<br>
Dennis<br>
>From <a href=3D"mailto:chris@kx.com">chris@kx.com</a> =C2=A0Sat Mar 16 18:0=
7:15 2013<br>
X-Google-DKIM-Signature: v=3D1; a=3Drsa-sha256; c=3Drelaxed/relaxed;<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 d=3D<a href=3D"http://google.com" target=3D"_bl=
ank">google.com</a>; s=3D20120113;<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 h=3Dx-received:mime-version:in-reply-to:referen=
ces:from:date:message-id<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0:subject:to:cc:content-type:x-gm-message-=
state;<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 bh=3D71wSAgAcVTdzFTENy6y2sp2imBZvf+9jhQW0rp71jP=
Y=3D;<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 b=3DeGl75j6NLKXrsc215ofg+M/iu7l7zNZg7RZalH1HRtK=
migN9LkBj95q9wMEMwGAOZy<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0kFX1bJpxpN0caUzV2doTW0kGqFOT06nRzlayQ2NQ+=
CQCgdVltydUtk1wGo4fsvuJXexs<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0El6W8344c06LDlUHWkOgsq4xRWR/S9zDbppc5X5v3=
lq5KrWkren32m54tzPc3gqG8LXz<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0v5iVcRNUELctkvKEz50DKeChFWJF5pTpD11XZ1snb=
UvXOM/mpp6EKWugEfbQPYbnse6x<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03dVc4tdO8/Yf6gRoYCV9tORXH30UeAeGWbgxxw2T/=
uyg75H48pnaFth1KHkj6XSOhQG4<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/Nlg=3D=3D<br>
X-Received: by 10.14.215.193 with SMTP id e41mr32601170eep.32.1363471627613=
;<br>
=C2=A0Sat, 16 Mar 2013 15:07:07 -0700 (PDT)<br>
MIME-Version: 1.0<br>
From: Chris Burke &lt;<a href=3D"mailto:chris@kx.com">chris@kx.com</a>&gt;<=
br>
Date: Sun, 17 Mar 2013 06:06:27 +0800<br>
Subject: Re: r to q<br>
To: Dennis Shasha &lt;<a href=3D"mailto:shasha@courant.nyu.edu">shasha@cour=
ant.nyu.edu</a>&gt;<br>
Cc: <a href=3D"mailto:shasha@cs.nyu.edu">shasha@cs.nyu.edu</a><br>
X-Gm-Message-State: ALoCoQle4s4kBZMfiAAg4Cxl1RjmFTYy1nVdmxFq3M+v6xZiA32vU2b=
lVFTwKr42ZA/VG/d87sNG<br>
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.1 (<a=
 href=3D"http://mx.cims.nyu.edu" target=3D"_blank">mx.cims.nyu.edu</a> [128=
.122.49.99]); Sat, 16 Mar 2013 18:07:13 -0400 (EDT)<br>
X-Virus-Scanned: clamav-milter 0.97.4 at mx<br>
X-Virus-Status: Clean<br>
X-Scanned-By: MIMEDefang 2.73 on 128.122.49.99<br>
<br>
--e89a8f647ac306b6a104d811fab5<br>
Content-Type: text/plain; charset=3DUTF-8<br>
<div><div class=3D"h5"><br>
Hi Dennis<br>
<br>
What version of OSX do you have?<br>
<br>
Chris<br>
<br>
On Sun, Mar 17, 2013 at 12:26 AM, Dennis Shasha &lt;<a href=3D"mailto:shash=
a@courant.nyu.edu">shasha@courant.nyu.edu</a>&gt;wrote:<br>
<br>
&gt;<br>
&gt; Dear Chris,<br>
&gt; In your mail you said:<br>
&gt;<br>
&gt; =3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
&gt; The attached was built with gcc 4.7 for R 2.15.3 on osx 10.8.2. This a=
llows<br>
&gt; R to connect to a q server from an R client, see examples in the zip.<=
br>
&gt; =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
&gt;<br>
&gt; Naively, I thought that I could use a slightly older mac os<br>
&gt; and R package without any special compilation effort.<br>
&gt; So, I tried the following<br>
&gt;<br>
&gt; q sp.q -p 5000<br>
&gt;<br>
&gt; R<br>
&gt; -- and then I tried dyn.load(&quot;qserver.so&quot;)<br>
&gt;<br>
&gt; but I get the following error:<br>
&gt;<br>
&gt; Error in dyn.load(&quot;qserver.so&quot;) :<br>
&gt; =C2=A0 unable to load shared object<br>
&gt; &#39;/Users/dennisshasha/q/smartiesinterconnect.d/qserver.so&#39;:<br>
&gt; =C2=A0 dlopen(/Users/dennisshasha/q/smartiesinterconnect.d/qserver.so,=
 6):<br>
&gt; Symbol not found: ___emutls_get_address<br>
&gt; =C2=A0 Referenced from: /Users/dennisshasha/q/smartiesinterconnect.d/q=
server.so<br>
&gt; =C2=A0 Expected in: /usr/lib/libSystem.B.dylib<br>
&gt; =C2=A0in /Users/dennisshasha/q/smartiesinterconnect.d/qserver.so<br>
&gt;<br>
&gt;<br>
&gt; So, the question is:<br>
&gt; what is the overall procedure? For example, should I put qserver.so<br=
>
&gt; somewhere<br>
&gt; else?<br>
&gt;<br>
&gt; Sorry for the elementary question.<br>
&gt; I&#39;ve never connected any language to q.<br>
&gt;<br>
&gt; Thanks,<br>
&gt; Dennis<br></div></div></blockquote></div>

--047d7b604fbaeb861004d81c49e8--

