[coral-announce] Changes to Coral's over-quota behavior

[Michael J. Freedman]

Hello,

After continued discussions on the coral-users list, we've slightly 
modified Coral's behavior when a hostname's "quota" is expired and Coral 
had previously returned an 403 FORBIDDEN error message.

Server operators who rely on Coral for their content distribution should 
make sure they read and understand this email.  Please send coral-users 
with any questions or further discussion.

Remember that there's 3 conditions by which any *one* Coral node will 
return an over-quota error:

   1. The node's hourly peak bandwidth has been exceeded (curr. 1.5 GB)
   2. The node's steady-state bandwidth has been exceeded (curr. 500 MB)
   3. The node has transmitted more than some limit for a SPECIFIC
      hostname; although other hostnames may continue to return
      succesfully.

Note that after a node has ran for a while, #1 no longer plays a role (as 
it's dominated by #2). #2 is designed to prevent a node's overuse 
(required by our node operators), while #3 attempts to provide fairness 
between different users of Coral give resource contention.

Now, instead of always returning a 403 message, the following occurs:

   -- If an error would be returned due to #1 or #2, the client is instead
      redirected back to the origin site, as we want to provide a better
      failover condition.

      Furthermore, as this node can't offer any further benefits in terms
      of content distribution for *anybody*, it unregisters itself from the
      indexing system for 30 minutes to recover some "useable" bandwidth
      and otherwise reduce the probability of such redirects when not
      strictly necessary.

   -- If a specific hostname is over quota (#3), the 403 error message is
      still returned BY DEFAULT.  (As if more stress is placed on a site
      which couldn't otherwise handle the load, Coral may have trouble
      fetching any further content from the site, e.g., it might go
      offline, thereby decreasing its overall availability.)

      However, servers can OVERRIDE this default behavior and cause Coral
      to redirect clients back to the origin server if they want to handle
      this additional load.

There are two main things for server operators to thus consider.

First, when Coral redirects a client back to the origin server, it appends 
a "?coral-no-serve" to the end of the URL, i.e.,

   http://www.example.net/foo --> http://www.example.net/foo?coral-no-serve

An unmodified server (i.e., not running any dynamic cgi script) will 
safely ignore this additional query string; it simply serves the file 
"foo".  So, basic servers will not require any changes to handle this 
query-string added by Coral.

However, if servers use Apache's mod_rewrite to redirect all non-Coral 
requests to Coral, they need to be careful to check that no query 
"?coral-no-serve" is present.  Consider the following message flow:

   client -> server:         http://example.net/
   server redirects client
   client -> coral:          http://example.net.nyud.net:8090/
   coral redirects client
   client -> server:         http://example.net/?coral-no-serve

Servers ignoring this query-string and redirecting the client *again* back 
to Coral would cause a loop.  To prevent this, add one line to your 
mod_rewrite rule (either in httpd.conf as root or in .htaccess in your 
local directory) to check for the query-string.

   RewriteEngine on
   RewriteCond %{HTTP_USER_AGENT} !^CoralWebPrx
   RewriteCond %{QUERY_STRING} !^coral-no-serve       # <-- NEW LINE
   RewriteRule ^/images/foo(.*)$
   http://foo.bar.nyud.net:8090/images/foo$1.html [R,L]

Finally, if you WANT to change the default behavior for condition #3 from 
returning a 403 error to redirecting clients back to the origin site, you 
should add a "X-Coral-Control: redirect-home" header to your HTTP 
responses.  To do this, add

   Header append X-Coral-Control "redirect-home"

to either httpd.conf (as root) or .htaccess (in your local directory). 
Note that you must have Apache compiled with mod_headers to perform this 
functionality.  For more instructions, please see Leo Laporte's email:

   http://www.cs.nyu.edu/pipermail/coral-users/2005-March/000229.html

If you use any other web servers, please feel free to either send 
instructions on what modifications you required, or add these instructions 
to our wiki at http://wiki.coralcdn.org/wiki.php/Main/Servers.

These changes have gone into effect across the network within the past few 
hours.  Please report any problems.  Special thanks for Sandy McArthur for 
helpful discussions.

Thanks,
--mike

-----
"Not all those who wander are lost"             www.michaelfreedman.org
"You make it fun; we'll make it run"                   www.coralcdn.org