[coral-announce] Recent, changes, bandwidth caps, module request, etc.

[Michael J. Freedman]

Hello,

A number of changes are being deployed into the Coral network since
the last announcement 6 weeks ago.  Some of these are currently being
tested on only a subset of the nodes; I wanted to send out an
announcement while we deploy to (a) give a "heads up", (b) welcome
suggestions / feedback, (c) look for possible contributors for an
automated flash-crowd-handling Apache module (#2 below).

Before the details, I just wanted to mention two new URLs for Coral:

   Coral website:  http://www.coralcdn.org/
   Coral wiki:     http://wiki.coralcdn.org/


1. Most importantly, we've begun deploying some bandwidth caps and
shaping algorithms.  Basically, Coral has grown in interest in the
past few months, and we're basically dominating the total traffic on
PlanetLab to levels that make some operators uneasy.  In the long
term, individuals stepping up and running Coral nodes will help this
process by increasing the total available bandwidth, but some type of
configuration parameters must still be there in those situations as
well.  So, we need to deal with some issues first-hand.

Because of this, Coral may return a 403 message that says basically
"Quota exceeded; try back later" on a per-node basis.  This decision
is calculated via three parameters:

   A. A node has some maximum peak bandwidth it is willing to
      send per hour (currently set at 1350 MB / hour)

   B. A node has some maximum steady-state bandwidth it is willing to
      send (currently set at 400 MB / hour)

   C. A node has some maximum steady-state bandwidth it is willing to
      send PER HOSTNAME (currently set at 100 MB / hour)

Steady-state are calculated via an exponentially-weight moving
average, taken over the past day.  (C) basically suggests that the
site may expect Coral to transmit an upper-bound of 250 GB per day
given the current deployment at 100-120 nodes.

We also play some additionally tricks that if we're actually hitting
the max per-node caps of (B), as opposed to the max per-hostname caps,
it's going to decrease the limits on per-hostname caps to achieve
greater fairness.

We're taken this approach with the following motivation in mind: Coral
is primarily designed to enable sites to weather flash crowds and
distribute recently-popular data.  It can't sustainably play the role
of a free, unlimited-bandwidth hosting service for an unlimited number
of servers, without severely degrading service for others and annoying
operators.  (We've seen one or two sites basically chew up > 50% of
the total traffic, over 1 TB / day).  Thus, the above design is
motivated by the goal of reducing steady-state bandwidth hogs, while
still enabling flash-crowds.  Comments?



2. With this in mind, I also have an idea for a easy Apache plugin
that can use a mixture of mod_bandwidth and mod_rewrite to
automatically detect when a server is currently seeing higher amounts
of load, and then, instead of just cutting off access via
mod_bandwidth, automatically redirecting to Coral.  I think this is
something people have asked for in the past.

This combines the benefit of allowing servers to continue to serve
traffic themselves when they can, but only falling back to Coral when
necessary.  (1) Given them greater control over their site (in terms
of logs, faster update to content if no expiry present, etc.); (2)
making better use of Coral's resources (both bandwidth and disk); (3)
given that Coral isn't currently useable by everybody (given port
8090), under high load, it's probably better for a site to be able to
reach 90% of its audience via Coral than 0% given traffic demands. If
anybody is interested in this, please contact me for more details.  I
think it would be fairly straight-forward given some existing modules
(and especially for anybody with prior experience with such).



3. Finally, I've converted all the programs to now use flexible
configuration files for all important variables, instead of a mix of
command-line and compile-time options.  This should make it easier for
people to download and run their own versions of the software; I know
of a few who have begun playing around with it themselves.

Some detailed changes below.

Thanks,
--mike

P.S.  As always, if an artistically-motivated person wishes to come up 
with some "Powered by Coral" logos, we'd sincerely appreciate any and 
recognition shall be yours!



Changes to corald
-----------------

A number of changes to improve the freshness of cached neighbor lists
in the DSHT.  This should hopefully improve ability for nodes to find
entries put () into system.

   - Fixed bug where stale references weren't getting removed from
     table, cause possible misses during lookup as "spread" during gets
     would be all dead, while puts placed earlier given dead spread.

   - Reduce timeout count on known node when act as server for node

   - Calculate RTT and TMO with TCP-style exponentially-weight moving 
average

   - Nodes only merge neighbors gossiped by others if they have also
     successfully contacted node in course of current lookup.

   - DropRate for cluster selection not being calculated correctly;
     temporarily dropped reliance on this metric


Changes to coralwebsrv
----------------------

Reperform Coral put on cached objects if fetched within past caching
period (2 hours).  Should reduce fetches from server for reasonably
(but not *very*) popular items.

Rewrite referer to not pass Coralized URL to stop some annoying
hotlink prevention mechanisms.  Only works sometimes -- suggestions
for further techniques to circumvent hotlink prevention would be
appreciated.

Synthesize robots.txt to always Disallow all

Disallow recursive request for nyud.net, searched to localhost, or DNS
resolution with directory search enabled.


Changes to coraldnssrv
----------------------

Query network separately for DNS and proxy servers.  Before, if DNS
servers also ran web proxies, just return those A records.
Unfortunatley, this meant load on servers *only* running web proxies
was next to zero.


New daemon:  probed
-------------------

Deploying new probed daemon that will monitor all HTTP requests from
clients and traceroute these clients to build network map for network
measurement / locality research.  Should only query a given IP address
at most once per hour per Coral host.

Thus, where before users may have seen TCP/ICMP probes to their DNS
server, you might not occassionally see ICMP probes to your client box
from where web request is originating.


-----
"Not all those who wander are lost."           www.michaelfreedman.org