HCP Upload Daemon

Document created by Thorsten Simons on Mar 9, 2017Last modified by Michael Ratner on Sep 18, 2017
Version 2Show Document
  • View in full screen mode

hcpupd is a daemon that automatically uploads files to HCP.

 

It's using the Linux inotify kernel subsystem to monitor a folder (aka *watchdir*), sending every file that is moved or written to it to

HCP, immediately. Folders created in the watchdir will be watched as well.

 

Features:

 

  • Two different upload modes:

1. Transfer the folder structure created in *watchdir* to HCP as it is

      • human-readable
      • performance-wise not the best possible solution
      • not tolerant against filename duplicates (except if the Namespace has Versioning enabled)

or

2. Obfuscate the folder structure on HCP by creating an UUID per file, used as filename as well as to construct a path to it

      • best possible ingest performance
      • 64k folders created at max. (to be precise: 256**2 + 256)
      • an unlimited number of hcpupd's can write into the same Namespace without the risk of filename conflicts
      • supports search for the original filename by adding an annotation to each file, which can be used by HCP's Metadata Query Engine, Hitachi Content Intelligence or any other indexer that is able to crawl a HCP Namespace
  • Optionally:
    • on start-up, upload existing files already stored in the watchdir
    • auto-delete files from *watchdir* after successful upload
    • auto-delete folders after the last file has been uploaded
    • Made to run as a Linux daemon (but can run in an interactive session as well)
    • Extended logging available, incl. log rotation

 

Documentation can be found at ReadTheDocs.

Attachments

    Outcomes