From simon@kx.com  Mon May 21 05:31:56 2012

Hi Dennis,

1. not directly, although if you've already got an old partition it's easier to work out where to split.
when taq volumes started going over 2B/day we did add some code to kdb+taq that sliced the data into 
multiple partitions - but that was at write time.

the trick is playing around with par.txt, spreading data over multiple locations by just replicating the date 
and then keeping A-M (say) in one place, N-Z another 

http://code.kx.com/wiki/Reference/pardottxt
and
http://code.kx.com/wiki/JB:KdbplusForMortals/kdbplus_database

the split can be done while the system is running by just adding new partitions and then forcing a reload of the 
directories when everything's finished with \l 

2. there's nothing special for this 
there's a q internal \r which is just the linux system mv, so it won't go across all file systems 
otherwise there's nothing special - just OS level mv/cp - with or without compression depending on the setup 

3. that's just the OS level rm, there's no special meta-data to be cleaned up 

some of the things like this are provided by the dbmaint.q script on code.kx 

http://code.kx.com/wiki/Contrib/dbmaintdotq#dbmaint.q

for you the most useful thing will likely be the code that quickly generates all the actual locations and 
allows you to manipulate those 

incidentally, with 3.0 there are UUIDs, which are a help with distributed systems 
less need for centralised "master lists" 

ireland .. yes, off tomorrow, the weather forecast isn't looking great .. 


On Friday, 18 May 2012 at 23:13, Dennis Shasha wrote:

> 
> Dear Simon,
> I'm trying to design a scaleable distributed system.
> This requires 
> 1. splitting partitions into subpartitions
> 2. copying a partition from one machine to another
> 3. deleting a partition
> 
> Do you know of any code for such things?
> Best,
> Dennis
> P.S. Have fun in Dublin. I'll be (unfortunately) in Scottsdale.



From listbox+trampoline+Q4+EFZ7+BkIojc-k4RGTs4Hm9W5HsA+BqUXUH@jeeves.archives.listbox.com  Wed May 23 08:07:02 2012
Received: from mx2.cims.nyu.edu (mx2.cims.nyu.edu [128.122.49.96])
	by mail.cims.nyu.edu (8.14.4+Sun/8.14.4) with ESMTP id q4NC71no026377
	for <shasha@mail.cims.nyu.edu>; Wed, 23 May 2012 08:07:01 -0400 (EDT)
Received: from b-lb-ob-quonix.listbox.com (b-lb-ob-quonix.listbox.com [208.72.237.5])
	by mx2.cims.nyu.edu (8.14.4+Sun/8.14.4) with ESMTP id q4NC6wOW005907
	for <shasha@cs.nyu.edu>; Wed, 23 May 2012 08:06:58 -0400 (EDT)
Received: from a-lb-ex-quonix.listbox.com (a-lb-ex-quonix.listbox.com [208.72.237.38])
	by b-lb-ob-quonix.listbox.com (Postfix) with ESMTP id 7C25DE22
	for <shasha@cs.nyu.edu>; Wed, 23 May 2012 08:06:58 -0400 (EDT)
Received: from localhost.localdomain (localhost [127.0.0.1])
	by a-lb-ex-quonix.listbox.com (Postfix) with ESMTP id 6DD9EDC8
	for <shasha@cs.nyu.edu>; Wed, 23 May 2012 08:06:58 -0400 (EDT)
Received: from b-lb-mx-sd.listbox.com (b-lb-mx-sd.listbox.com
 [74.115.168.28]) by a-lb-arc-quonix.listbox.com (Postfix) with ESMTP id
 EDDA34C8EA for <k4@listbox.com.email.enqueue.archive.listbox.com>; Wed, 23
 May 2012 08:05:13 -0400 (EDT)
Received: from localhost.localdomain (localhost [127.0.0.1]) by
 b-lb-mx-sd.listbox.com (Postfix) with ESMTP id DEA3798F0 for
 <k4@listbox.com.email.enqueue.archive.listbox.com>; Wed, 23 May 2012 08:05:13
 -0400 (EDT)
Received: from dubmail.firstderivatives.com (dubmail.firstderivatives.com
 [80.169.143.228]) by b-lb-mx-sd.listbox.com (Postfix) with ESMTP id E0C3298EF
 for <k4@listbox.com>; Wed, 23 May 2012 08:05:09 -0400 (EDT)
Received: by dubmail.firstderivatives.com (Postfix, from userid 106) id
 9B1BE3F45F6; Wed, 23 May 2012 12:05:08 +0000 (UTC)
Received: from dubmail.firstderivatives.com (localhost [127.0.0.1]) by
 dubmail.firstderivatives.com (Postfix) with ESMTP id 877693F45DE for
 <k4@listbox.com>; Wed, 23 May 2012 12:05:08 +0000 (UTC)
X-Virus-Scanned: by SpamTitan at example.com
Received: from dubmail02.firstderivatives.com (unknown [10.152.25.70]) by
 dubmail.firstderivatives.com (Postfix) with ESMTP id CCC2D3F45F6 for
 <k4@listbox.com>; Wed, 23 May 2012 12:05:02 +0000 (UTC)
Received: from mail2.firstderivatives.com ([10.18.1.12]) by
 dubmail02.firstderivatives.com with Microsoft SMTPSVC(6.0.3790.4675); Wed, 23
 May 2012 13:05:02 +0100
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Subject: RE: [k4] HDB Compression
Date: Wed, 23 May 2012 13:04:39 +0100
Message-ID: <4F773F4191C4144CB234DDF7FF75A91FE2E77E@MAIL02.FD.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: [k4] HDB Compression
Thread-Index: Ac04cn4NfUJxxG0qQ6WJRQwBLcds/wAaGarQAABVdoY=
References: <21DD8CDC8F739748A27B5541BD029F64029E3BD9@NSTMC101PEX1.ubsw.net>
 <49A56BCD-5FC6-4382-8C37-E342A51B5FF5@kx.com>
 <21DD8CDC8F739748A27B5541BD029F64029E3BF2@NSTMC101PEX1.ubsw.net>
From: "Ryan Hamilton" <rhamilton@firstderivatives.com>
To: "k4" <k4@listbox.com>
X-OriginalArrivalTime: 23 May 2012 12:05:02.0591 (UTC)
 FILETIME=[489E58F0:01CD38DC]
Reply-To: k4@listbox.com
List-ID: <k4@listbox.com>
X-Listbox-List-ID: 1080 <k4@listbox.com>
List-Software: listbox.com v2.0
List-Help: <http://www.listbox.com/subscription-help.html>
Precedence: list
List-Unsubscribe: <mailto:unsubscribe-k4@listbox.com>,
 <https://www.listbox.com/unsubscribe/?member_id=901033&id_secret=901033-6a517507&post_id=20120523080515:8D284206-A4CF-11E1-93B3-81E6F56E47B0>
List-Subscribe: <mailto:subscribe-k4@listbox.com>,
 <http://www.listbox.com/subscribe/?list_id=1080>
X-Listbox-Post-Id: 20120523080515:8D284206-A4CF-11E1-93B3-81E6F56E47B0
X-Listbox-Original-Sender: "Ryan Hamilton" <rhamilton@firstderivatives.com>
Errors-To:  listbox+trampoline+Q4+EFZ7+BkIojc-k4RGTs4Hm9W5HsA+BqUXUH@jeeves.archives.listbox.com
X-Scanned-By: MIMEDefang 2.73 on 128.122.49.96
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.6 (mx2.cims.nyu.edu [128.122.49.96]); Wed, 23 May 2012 08:07:01 -0400 (EDT)
Status: R
Content-Length: 3545

Util here should help, see example at bottom.
http://code.kx.com/wsvn/code/contrib/simon/compress/cutil.q


-----Original Message-----
From: ryan.minogue@ubs.com [mailto:ryan.minogue@ubs.com]
Sent: Wed 23/05/2012 07:59
To: k4
Subject: RE: [k4] HDB Compression
=20
Ah I'm only on 2.7, so guess I'll have to compress after the fact. Can
that be done on the whole partition at once or do I have to go table by
table?

________________________________

From: Charles Skelton (Kx) [mailto:charlie@kx.com]=20
Sent: Tuesday, May 22, 2012 7:25 PM
To: k4
Subject: Re: [k4] HDB Compression


Which version/release are you using?
.z.zd is a 2.8 feature

On May 22, 2012, at 11:53 PM, <ryan.minogue@ubs.com> wrote:



	Is there a simple way to get my RDB to automatically compress
files when it writes down to the HDB at EOD? I tried setting .z.zd in
r.q but that didn't seem to have any effect.

k4 | Modify <https://www.listbox.com/member/?&>  Your Subscription=20=09

	Visit our website at http://www.ubs.com
=09
	This message contains confidential information and is intended
only=20
	for the individual named.  If you are not the named addressee
you=20
	should not disseminate, distribute or copy this e-mail.  Please=20
	notify the sender immediately by e-mail if you have received
this=20
	e-mail by mistake and delete this e-mail from your system.
=09=20=20=20
	E-mails are not encrypted and cannot be guaranteed to be secure
or=20
	error-free as information could be intercepted, corrupted, lost,

	destroyed, arrive late or incomplete, or contain viruses.  The
sender=20
	therefore does not accept liability for any errors or omissions
in the=20
	contents of this message which arise as a result of e-mail
transmission.=20=20
	If verification is required please request a hard-copy version.
This=20
	message is provided for informational purposes and should not be

	construed as a solicitation or offer to buy or sell any
securities=20
	or related financial instruments.
=09
=09
	UBS reserves the right to retain all messages. Messages are
protected
	and accessed only in legally justified cases.
=09
	-------------------------------------------
	k4
	Modify Your Subscription: https://www.listbox.com/member/?&
=09

k4 | Modify
<https://www.listbox.com/member/?&
1bcf6fa>  Your Subscription=20=09


***************************************************************************=
********************
This email, its contents and any files attached are a confidential communic=
ation and are
intended only for the named addressees indicated in the message.

If you are not the named addressee or if you have received this email in er=
ror, you may not,
without the consent of First Derivatives, copy, use or rely on any informat=
ion or attachments in any way.
Please notify the sender by return email and delete it from your email syst=
em.

Unless separately agreed, First Derivatives does not accept any responsibil=
ity for the accuracy or
completeness of the contents of this email or its attachments. Please note =
that any views,
opinion or advice contained in this communication are those of the sending =
individual
and not those of First Derivatives and First Derivatives shall have no liab=
ility whatsoever in relation to
this communication (or its content) unless separately agreed.
***************************************************************************=
********************


-------------------------------------------
k4
Modify Your Subscription: https://www.listbox.com/member/?member_id=3D90103=
3&id_secret=3D901033-670b4995

