org.supermind.crawl
Class MD5Persister

java.lang.Object
  extended by org.supermind.crawl.util.MapFilePersister<org.apache.nutch.io.MD5Hash,org.apache.nutch.io.NullWritable>
      extended by org.supermind.crawl.util.MD5Persister

public class MD5Persister
extends MapFilePersister<org.apache.nutch.io.MD5Hash,org.apache.nutch.io.NullWritable>

Saves MD5Hash values to a MapFile.

Field Summary
 
Fields inherited from class org.supermind.crawl.MapFilePersister
buffer, maxBufferSize, sorter, tmpWriter
 
Constructor Summary
MD5Persister()
           
 
Method Summary
 void add(org.apache.nutch.io.MD5Hash hash)
           
 boolean contains(org.apache.nutch.io.MD5Hash hash)
           
protected  org.apache.nutch.io.WritableComparator getKeyComparator()
          Get comparator for MapFile key class.
 org.apache.nutch.io.WritableComparable getKeyInstance()
          Return a new instance of the key.
protected  java.lang.Class<? extends org.apache.nutch.io.WritableComparable> getMapFileKeyClass()
          Get key class.
 java.util.Comparator getTypeComparator()
          Get comparator for type.
 void writeBufferToTmp()
          Write contents of buffer to tmpfile.
 
Methods inherited from class org.supermind.crawl.MapFilePersister
add, close, flushToDisk, getMapFileValueClass, getValueInstance, init, initTmpWriter, setMapdir, setNfs, setOverwrite, setTmpfile
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

MD5Persister

public MD5Persister()
Method Detail

add

public void add(org.apache.nutch.io.MD5Hash hash)
         throws java.io.IOException
Throws:
java.io.IOException

contains

public boolean contains(org.apache.nutch.io.MD5Hash hash)
                 throws java.io.IOException
Throws:
java.io.IOException

getKeyComparator

protected org.apache.nutch.io.WritableComparator getKeyComparator()
Description copied from class: MapFilePersister
Get comparator for MapFile key class.

Specified by:
getKeyComparator in class MapFilePersister<org.apache.nutch.io.MD5Hash,org.apache.nutch.io.NullWritable>
Returns:

getKeyInstance

public org.apache.nutch.io.WritableComparable getKeyInstance()
Description copied from class: MapFilePersister
Return a new instance of the key.

Specified by:
getKeyInstance in class MapFilePersister<org.apache.nutch.io.MD5Hash,org.apache.nutch.io.NullWritable>
Returns:

getMapFileKeyClass

protected java.lang.Class<? extends org.apache.nutch.io.WritableComparable> getMapFileKeyClass()
Description copied from class: MapFilePersister
Get key class.

Specified by:
getMapFileKeyClass in class MapFilePersister<org.apache.nutch.io.MD5Hash,org.apache.nutch.io.NullWritable>
Returns:

getTypeComparator

public java.util.Comparator getTypeComparator()
Description copied from class: MapFilePersister
Get comparator for type.

Specified by:
getTypeComparator in class MapFilePersister<org.apache.nutch.io.MD5Hash,org.apache.nutch.io.NullWritable>
Returns:

writeBufferToTmp

public void writeBufferToTmp()
                      throws java.io.IOException
Description copied from class: MapFilePersister
Write contents of buffer to tmpfile. Subclasses should use MapFilePersister.tmpWriter to do this.

Specified by:
writeBufferToTmp in class MapFilePersister<org.apache.nutch.io.MD5Hash,org.apache.nutch.io.NullWritable>
Throws:
java.io.IOException