We need to ensure we're sane when dealing with larger files
requiring multiple reads.

This matches the buffer size used by replication, and showed a
performance increase when timing the 100M large file test in
t/40-httpfile.t

With the following patch, I was able to note a ~46 -> ~27s
time difference with both MD5 methods using this change
to increase buffer sizes.

  --- a/t/40-httpfile.t
  +++ b/t/40-httpfile.t
  @@ -125,5 +125,12 @@ $expect = $expect->digest;
   @paths = $mogc->get_paths("largefile");
   $file = MogileFS::HTTPFile->at($paths[0]);
   ok($size == $file->size, "big file size match $size");
  +use Time::HiRes qw/tv_interval gettimeofday/;
  +
  +my $t0;
  +$t0 = [gettimeofday];
   ok($file->md5_mgmt(sub {}) eq $expect, "md5_mgmt on big file");
  +print "mgmt ", tv_interval($t0), "\n";
  +$t0 = [gettimeofday];
   ok($file->md5_http(sub {}) eq $expect, "md5_http on big file");
  +print "http ", tv_interval($t0), "\n";

Base64 requires further escaping for our tracker protocol which
gets ugly and confusing.  It's also easier to interact/verify
with existing command-line tools using hex.

We need a place to store mappings for various checksum
types we'll support.

This is needed to wire up checksums to classes.

Digest::MD5 and Digest::SHA1 both support the same API for
streaming data for the calculation, so we can validate our
content as we stream it.

Checksum usage will be decided on a per-class basis.

This branch is now rebased against my latest clear_cache
which allows allows much faster metadata updates for testing.

Helps me keep my head straight.

This can come in handy.

We'll use the "Digest" class in Perl as a guide for this.
Only MD5 is officially supported.

However, this *should* support SHA-(1|256|384|512) and it's easy
to add more algorithms.

We can now:
* enable checksums for classes
* save client-provided checksums to the database
* verify them on create_close
* read them in file_info

we need to be able to both enable and disable checksuming for a class

This returns undef if a checksum is missing for a class,
and a MogileFS::Checksum object if it exists.

replication now lazily generates checksums if they're not
provided by the client (but required by the storage class).

replication may also verify checksums if they're available
in the database.

replication now sets the Content-MD5 header on PUT requests,
in case the remote server is capable of rejecting corrupt
transfers based on it

replication attempts to verify the checksum of the freshly
PUT-ed file.

TODO: monitor will attempt "test-write" with mangled Content-MD5
      to determine if storage backends are Content-MD5-capable
      so replication can avoid reading checksum on destination

This functionality (and a server capable of rejecting bad MD5s)
will allow us to skip an expensive MogileFS::HTTPFile->digest
request at replication time.

Also testing with the following patch to Perlbal:

  --- a/lib/mogdeps/Perlbal/ClientHTTP.pm
  +++ b/lib/mogdeps/Perlbal/ClientHTTP.pm
@@ -22,6 +22,7 @@ use fields ('put_in_progress', # 1 when we're currently waiting for an async job
             'content_length',  # length of document being transferred
             'content_length_remain', # bytes remaining to be read
             'chunked_upload_state', # bool/obj:  if processing a chunked upload, Perlbal::ChunkedUploadState object, else undef
+            'md5_ctx',
             );

 use HTTP::Date ();
@@ -29,6 +30,7 @@ use File::Path;

 use Errno qw( EPIPE );
 use POSIX qw( O_CREAT O_TRUNC O_WRONLY O_RDONLY ENOENT );
+use Digest::MD5;

 # class list of directories we know exist
 our (%VerifiedDirs);
@@ -61,6 +63,7 @@ sub init {
     $self->{put_fh} = undef;
     $self->{put_pos} = 0;
     $self->{chunked_upload_state} = undef;
+    $self->{md5_ctx} = undef;
 }

 sub close {
@@ -134,6 +137,8 @@ sub handle_put {

     return $self->send_response(403) unless $self->{service}->{enable_put};

+    $self->{md5_ctx} = $hd->header('Content-MD5') ? Digest::MD5->new : undef;
+
     return if $self->handle_put_chunked;

     # they want to put something, so let's setup and wait for more reads
@@ -421,6 +426,8 @@ sub put_writeout {

     my $data = join("", map { $$_ } @{$self->{read_buf}});
     my $count = length $data;
+    my $md5_ctx = $self->{md5_ctx};
+    $md5_ctx->add($data) if $md5_ctx;

     # reset our input buffer
     $self->{read_buf}   = [];
@@ -460,6 +467,17 @@ sub put_close {

     if (CORE::close($self->{put_fh})) {
         $self->{put_fh} = undef;
+
+        my $md5_ctx = $self->{md5_ctx};
+        if ($md5_ctx) {
+            my $actual = $md5_ctx->b64digest;
+            my $expect = $self->{req_headers}->header("Content-MD5");
+            $expect =~ s/=+\s*\z//;
+            if ($actual ne $expect) {
+                return $self->send_response(400,
+                    "Content-MD5 mismatch, expected: $expect actual: $actual");
+            }
+        }
         return $self->send_response(200);
     } else {
         return $self->system_error("Error saving file", "error in close: $!");

Rereading a large file is expensive.  If we can monitor
and observe our storage nodes for MD5 rejectionability, we
can rely on that instead of having to have anybody reread
the entire file to calculate its MD5.

Only the fsck part remains to be implemented... And I've never
studied/used fsck much :x

Stale rows are bad.

TODO: see if we can use LWP to avoid mistakes like this :x

Fsck behavior is based on existing behavior for size mismatches.
size failures take precedence, since it's much cheaper to verify
size match/mismatches than checksum mismatches.

While checksum calculations are expensive and fsck is already
parallel, so we do not parallelize checksum calculations on
a per-FID basis.

It reads more easily this way, at least to me.

I'll be testing checksum functionality on my home installation
before testing it on other installations, and I run SQLite at
home.

ref: http://www.sqlite.org/lang_altertable.html

We need to ensure the worker stays alive during MD5
generation, especially on large files that can take
many seconds to verify.

This special-cases "NONE" for no hash for our users.

We don't actually use the BLOB type anywhere, as checksums
are definitely not "L"(arge) objects.

The timeout comparison is wrong and causing ping_cb to never
fire.  This went unnoticed since I have reasonably fast disks
on my storage nodes and the <$sock> operation was able to
complete before being hit by a watchdog timeout.

Enabling this setting allows fsck to checksum all replicas on
all devices and report any corrupted copies regardless of
per-class settings.

This feature is useful for determining if enabling checksums on
certain classes is necessary and will also benefit users who
cannot afford to store checksums in the database.

MD5 is faster than SHA1, and much faster than any of the SHA2
variants.  Given the time penalty of fsck is already high with
MD5, prevent folks from shooting themselves in the foot with
extremely expensive hash algorithms.

Unlike the setting it replaces, this new setting can be used to disable
checksumming entirely, regardless of per-class options.

fsck_checksum=(class|off|MD5)

class - is the default, fsck based on per-class hashtype
off - skip all checksumming regardless of per-class setting
MD5 - same as the previous fsck_auto_checksum=MD5

MD5 is I/O-intensive, and having fsck request MD5s concurrently
ends up causing I/O contention on rotational drives with high
seek latency.  So limit fsck MD5 requests to a single job per
device.

The digest path relies on having a known file size to calculate
the MD5 timeout, so save an HTTP HEAD request since we always
check file sizes in fsck before we checksum the file.

Optimized SHA-1 implementations aren't significantly slower than
MD5 and some folks (e.g. Tomas Doran) may already have SHA-1 in
place for their data.

A liberally licensed, GPL-compatible collection of SHA-1
primitives is available from one of the OpenSSL developers:

  http://www.openssl.org/~appro/cryptogams/

It would be nice to allow the Perl Digest module to
transparently take advantage of architecture-specific
optimizations.

Note there is no standardized equivalent to the HTTP Content-MD5
header/trailer for any of the SHA variants, so verification for
replication/uploads may take significantly longer.

Requested-by: Tomas Doran

$NAME is potentially ambiguous and $HASHTYPE matches the
database column name.

Fsck would print "Status: N / 0 " if it's never been started before. Now
internally finds the max(fid) on its own.

if fsck_checksum was set to off, it would ignore the checksums deep in the
code, but would still attempt to "fix" the fids each time, which runs far more
code and UPDATE's each fid's devcount even if you tell it not to.

now it does what it should. however FSCK with checksums enabled will still
UPDATE devcount on each check.

Changelog diff is:

diff --git a/CHANGES b/CHANGES
index 770e518..5b59d7f 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,3 +1,12 @@
+2012-03-30: Release version 2.60
+
+   * Fix fsck status when running for the first time (dormando <[email protected]>)
+
+   * Checksum support (Major update!) (Eric Wong <[email protected]>)
+     See doc/checksums.txt for an overview of how the new checksum system
+     works. Also keep an eye on the wiki (http://www.mogilefs.org) for more
+     complete documentation in the coming weeks.
+
 2012-02-29: Release version 2.59

    * don't make SQLite error out on lock calls (dormando <[email protected]>)

Buffering log output in memory makes it difficult to view debug
and error output during development.  Since MogileFS does not
write to stdout frequently, there should be no noticeable
performance loss from this change.

This also prevents mangling of TAP output which caused
test failures if DEBUG=1 is set.

I noticed that attempting to delete a domain with classes returns
an unhelpful "Operation failed" error message.

There's no need to broadcast changes to other workers if there
are no changes.  Since HTTP servers rarely (if ever) change
their ability to toggle Content-MD5 rejection, this was causing
needless wakeups in every monitor round.

Tested by running mogilefsd with DEBUG=1 and using toggling
Content-MD5 rejection in mogstored + perlbal 1.80 via:

	SET mogstored.enable_md5 = (0|1)

to the mgmt port while watching syslog output.

Noticed-by: dormando

Before we make changes to the fsck code, we should ensure
we don't break existing use cases.

Behavior I'm uncertain about is documented with "XXX".

The LWP::UserAgent module found in my Debian 6.0 installation
does not have a "delete" convenience wrapper.

No point in using DBI directly if a task can be done directly
via the MogileFS::Store API.

Noticed-by: dormando

This was leftover when I was monitoring the test with DEBUG=1 :x

* ensure fsck can handle a stall from an unresponsive mogstored

* ensure over-replicated files are cleaned up

* ensure fsck can work correctly with dead devices if it beats
  reaper to an FID

A BCNT error is more descriptive than a generic POVI entry
and more accurately reflects the change made to an FID entry.

This also removes a dependency from /using/ the devcount column
and simplifies the code.  The devcount column remains invaluable
and informative to users, but MogileFS should not trust it for
making decisions when it has access to the file_on table.

We need to ensure fsck can resume and returns sane
stats output.

Hopefully this can eliminate some random test failures.

After resetting, fsck_highest_fid_checked ends up at zero.

We checked the incorrect return value, so the second
mogstored failing would've gone unnoticed.

These race conditions were causing this test fail occasionally.
These test failures were more common on SQLite and Postgres,
but not unheard of when using MySQL.

Some of these race conditions were due to fsck/job_master not
receiving settings changes in time, so we now resort to killing
existing processes and forcing them to reload.

Whenever an FID is unfixable, be sure to update devcount (to
zero) to easily inform the user via mogstats.  If the FID
magically reappears in the future, the desperate fallback mode
will still find the file.

fix_fid(): we no longer rely blindly update devcount on every
call.  This is important because we call fix_fid() on checksum
checks regardless, and devcount updates entail unnecessary
updates to the `file' table.

While we're at it, consolidate the places where we check the
skip_devcount flag and log bad devcounts.

Instead of blindly sleeping, we can '!watch' through the tracker
and detect the the error message fsck sends.

This is a simpler implementation and lets us be notified of
worker death (and pending replacement) as soon the tracker
notices it.

FIDs may be created while fsck is running, causing "mogadm fsck
status" to report completion above 100% (and thus confusing
users).  Stopping fsck when it reaches fsck_fid_at_end (set to
MAX(fid) at fsck startup) prevents this.

This change should also have a pleasant side effect of reducing
contention with replicate workers on newly-uploaded FIDs.

ref: http://code.google.com/p/mogilefs/issues/detail?id=50

MogileFS::Server and Perlbal both ignore SIGPIPE for
us.  So there's no need to ever ignore it for socket
writes in HTTPFile, either.

Use the replicate lock here to prevent an DevFID from being
orphaned by a replicate process.  This prevents orphaned
requests if a user issues a delete request on a file while
replication worker is copying it.

This avoids an uninitialized value warning from Perl when
choosing a value for the deprecated listen_jobs value.  Neither
the child nor the parent processes are capable of handling undef
values from :set_config_from_*.

Most of this was already nuked in the following commits:
ebf8a5a
3db8a84

Unused since commit 0be2f97
when the old drain/rebalance code was dropped.

Based on my reading of the code, the conditional assignments
of $hostip, $get_port, $devid, $timeout, and $url are needless.

Keeping track of explicit test counts causes needless merge
conflicts.  done_testing() is sufficient to note test
completeness and detect failures.

may annoy centos5 users.

Unused since commit 6c23c9d
("make fsck worker distributed").  Since this always seemed
fsck-specific, it's also unlikely plugins are using this.

Queries are computed in parallel and therefore replies are not in
the right order. This patch adds an optional callid parameter so
a caller can match the replies back to the queries. ERR lines
return the callid as 3rd parameter. If the callid parameter is
missing then the Protocol is the same as before.

Example:
GET_PATHS callid=1
ERR no_domain No+domain+provided 1
GET_DOMAINS callid=1
OK domains=0&callid=1

This is adapted from the Postgres implementation, but
since SQLite runs on one machine, we can use kill 0
to detect if a process is got nuked before it could
release a lock.

Since all Store implementations implement get_lock/release_lock,
we can safely share the same implementation of both
should_begin_replication_fidid() and note_done_replicating().

For rare SQLite setups, drop locks after 3600s regardless of the
hostname of the lock holder.  This can work around weird setups
that change hostnames (frequently) or share SQLite DBs over NFS.

The devcount of a newly uploaded file is always 1, so
we do not need another set of trips to the DB to set
this in the file row.

Specifying "alivetypo" as the host status would cause mogilefs to implode.

This possible fix could also be hiding another bug, but the
original test ordering was suspect...

Changelog diff is:

diff --git a/CHANGES b/CHANGES
index 5b59d7f..de3ba9b 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,3 +1,33 @@
+2012-05-18: Release version 2.61
+
+   * fix issue #57 by Pyry and Eric (dormando <[email protected]>)
+     (mogadm host status sometimes allowed typos)
+
+   * avoid unnecessary devcount update in create_close (Eric Wong <[email protected]>)
+
+   * sqlite: implement locking via tables (Eric Wong <[email protected]>)
+
+   * worker/query: Add optional callid parameter (Gernot Vormayr <[email protected]>)
+     (allows command pipelining)
+
+   * delete: prevent orphan files from replicator race (Eric Wong <[email protected]>)
+
+   * fsck: prevent running over 100% completion (Eric Wong <[email protected]>)
+
+   * fsck: cleanup and reduce unnecessary devcount updates (Eric Wong <[email protected]>)
+
+   * fsck: update devcount, forget devs on unfixable FIDs (Eric Wong <[email protected]>)
+
+   * fsck: log bad count correctly instead of policy violation (Eric Wong <[email protected]>)
+
+   * tests: add test for fsck functionality (Eric Wong <[email protected]>)
+
+   * monitor: only broadcast reject_bad_md5 on change (Eric Wong <[email protected]>)
+
+   * worker: delete_domain returns has_classes error (Eric Wong <[email protected]>)
+
+   * log: enable autoflush for stdout logging (Eric Wong <[email protected]>)
+
 2012-03-30: Release version 2.60

    * Fix fsck status when running for the first time (dormando <[email protected]>)

Schema upgrade needs to use Pg-specific column types for the v15 upgrade
adding class.hashtype. Only CREATE TABLE is auto-converted where
possible, not ALTER TABLE.

Signed-off-by: Robin H. Johnson <[email protected]>

Changelog diff is:

diff --git a/CHANGES b/CHANGES
index b526b6a..6333f9f 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,3 +1,9 @@
+2012-05-19: Release version 2.62
+
+   * Critical bugfix for a compilation error (dormando, reported by Robin)
+
+   * Fix for upgrading a Postgres install for checksums (Robin * H. Johnson <[email protected]>)
+
 2012-05-18: Release version 2.61

    * fix issue #57 by Pyry and Eric (dormando <[email protected]>)

commit f54e6fd botched
the ordering of parameters when updating the file table.

Changelog diff is:

diff --git a/CHANGES b/CHANGES
index 6333f9f..07790f3 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,3 +1,8 @@
+2012-05-29: Release version 2.63
+
+   * Critical bugfix for Postgres users introduced by 2.61. New file uploads
+     would fail. (noticed by robin H. Johnson, fixed by Eric Wong)
+
 2012-05-19: Release version 2.62

    * Critical bugfix for a compilation error (dormando, reported by Robin)

Host don't define readability/writability themselves, so the
Host::should_get_new_files sub is renamed to "alive" and
Device->can_read_from respects host status.

Also, queryworker now skips down/dead hosts in cmd_get_paths.

ref: http://code.google.com/p/mogilefs/issues/detail?id=46

The observed unreachable state of the host should always supercede the
observed state of the device.  This is already the case with
observed_writeable, but not with observed_readable nor
observed_unreachable.

The monitor worker does not (and should not, to save bandwidth) update
states of all devices when a host goes down.

should_read_from() should replace all uses of can_read_from() in
non-Monitor workers.  This avoids the overhead of needlessly
rechecking devices either the monitor or user marked down.

This simplifies queryworker logic a bit.

Avoid needlessly attempting connections for checking files on
host/devices the monitor (or user) marked as unreadable.

This also makes the Fsck->size_on_disk function redundant.

URLs pointing to devices set to drain are undesirable.
Files may disappear off draining devices immediately
after we've queried the file_on table and invalidate
the paths the client sends us.

This is mainly to prevent bugs like the fix in
commit ac5534a
from popping up again.

This subroutine has been unused since MogileFS 2.52
commit 18a40d2
("Throw out old HTTPFile->size code and use LWP")

We cannot break existing case-insensitive behavior for list_keys
right now, even if it's broken.  This means SQLite/MySQL will
use case-insensitive LIKE statements for list_keys and Postgres
remains case-sensitive.

Enabling this boolean will make the "after" and "prefix" params
of "list_keys" behave case-sensitively.

If this setting is /not/ enabled, clients will hit
after_mismatch errors when iterating through keys if they are
using an uppercase "prefix" argument and a subsequent list_keys
is called with an "after" that only matches case-insensitively.

If unset, this defaults to false (0) to match historical (buggy)
behavior.  Historical behavior is preserved (even if broken) as
users with small namespaces may rely on case-insensitive
matching.

Postgres users are not affected by this change, as the LIKE
operator in Postgres is always case-sensitive.

This change is tested on all three databases: Postgres, MySQL,
and SQLite.

Device state reporting functions should respect whatever the
underlying Host state is.

Because memcache TTL is now user configurable, data in memcached might
be valid for a long time, and as such invalid paths might be returned.

It would be possible to populate memcache, instead of just removing. But
it might be wasteful when a device is marked as dead, those replicated
fids might not need to be in memcached.

There is still one TODO left. If someone modifies mindevcount and runs
FSCK, then the mappings might become incorrect, but I reasoned that it
would be rather rare.

Changelog diff is:

diff --git a/CHANGES b/CHANGES
index 07790f3..c552089 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,3 +1,15 @@
+2012-06-21: Release version 2.64
+
+   * Delete memcache data when we replicate fids (Pyry Hakulinen <[email protected]>)
+
+   * implement "case_sensitive_list_keys" server setting (Eric Wong <[email protected]>)
+
+   * get_paths: deprioritize devs in "drain" state (Eric Wong <[email protected]>)
+
+   * make marking a host down cause devices to act as down (Eric Wong <[email protected]>)
+
+   * monitor skips hosts marked dead or down (Eric Wong <[email protected]>)
+
 2012-05-29: Release version 2.63

    * Critical bugfix for Postgres users introduced by 2.61. New file uploads

This makes it easier to test mock or alternative iostat
implementations.  This can be used for emulating the
iostat output on other platforms.