Back to the top of the comp.arch.storage FAQ.


3. Hierarchical Storage Management

HSM systems transparently migrate files from disk to optical disk and/or magnetic tape, usually robotically accessible. Then when files are accessed by a user, they transparently move them back to disk.

Watch for maximum file size limitations, sometimes limited by the size of the media used, sometimes by the server's OS, and sometimes neither.

Some offer integrated backup. Some will manage multiple copies of files for data reliability.

Some offer integrated migration from other systems (ie, file servers and/or workstations) to the central location disks, then to the central location robotics. This generally requires changes to the on-disk file system format on the migration clients.

An item to watch for is that the file management may be exactly like Unix -- that is, all files appear to be online, and once they're deleted, they're gone forever, even though the data may still be on tape.

All of the subsections here are Unix-compatible (in various flavors) unless indicated otherwise.

Additional Information:

See also _DEC Professional_, February 1993, Page 40 and _Client/Server Today_, Dec. '94, p. 60.

The System-Managed Storage Guide by Howard W. Miller, $225 for first copy, $75 for additional copies for same company available from The Information Technology Institute, 136 Orchard Street, Byfield, Massachusetts, 01922-1605.

(stith@fnal.gov)

Thomas Woodrow did an evaluation of NAStore, FileServ, DMF and Unitree in 1993. It can be obtained through www.nas.nasa.gov or the Proc. 3rd NASA Goddard Conference on Mass Storage Systems and Technologies, Oct. 1993, pp. 187--216. Somewhat dated now but excellent methodology for comparing HSMs.


3.1. Unitree {Brief}

The uncle of UNIX HSMs. Developed primarily at Lawrence Livermore National Laboratories. Commercialized by a company called DISCOS, then sold to OpenVision. UniTree was sold to UniTree Software in December, 1994. See www.unitree.com.

Many versions exist on different hardware platforms, including a National Storage Lab (NSL) UniTree commercialized by IBM - Fed Systems. It's also available on SGI, Convex, and Amdahl hardware, at least.

See also "Epoch vs Unitree" below

For Convex, try


Jim Wilson
214-497-3085
jrwilson@convex.com
Business Development
Data Management Applications
Convex Computer Corporation

For most other platforms, call Open Vision at (800)223-OPEN or (510)426-6400.

New info:


The latest release of UniTree, V1.9.1, has the following changes:
  • Available directly from UniTree Software Inc.
  • Support DEC, HP-UX, SGI, Sun
  • GUI(Tcl/Tk) tools for installation and administration
  • New name database structure
  • Common Message Logger
  • Parallel Migration and Staging
  • Multiple Storage Hierarchy (Optical/Tape)
  • FTP performance improvements (Read/Write 20MBs/16MBs)*
  • NFS performance improvements (Read/Write 3.5MBs/2MBs)*
  • Rule-based dynamic migration
  • Support for new robots (e.g., STK 97xx)
  • Support >2GB disk partitions on Sun
  • 64K File Families
  • Configurable media and drive types
  • Departmental File Server Configuration
  • Compatible with most backup software (Legato, CAM, SMArch) Demo copy available for download from web site: www.unitree.com New resellers in Asia, Europe, Australia * Measured on a dual cpu Sun Ultra3000 with 256MB and 10 disks -- Francis Kim Phone: (510) 833-3460 Director of Sales and Marketing FAX: (510) 833-9345 Unitree Software, Inc. e-mail: francis@unitree.com 11875 Dublin Blvd. Suite A200E WWW: http://www.unitree.com Dublin, Ca. 94568

  • 3.1.1. Epoch vs Unitree

    (Note: this evaluation is old, and should be taken with a grain of salt. rdv, 3/96)

    (6/93) We just bought both last year. We bought an Epoch I with the 20 GB EO and 327 GB worm. We will be upgrading it to an Epoch II soon. We also bought Unitree from Titan to run on a Silicon Graphics server and hook up to the STK 3480 silo. We hope to add more silos eventually.

    Unitree is licensed based on storage capacity while Epoch is not. There may be an exception to this - STK just began reselling Epoch as the front end for their silos and I'm not sure how they handle licensing.

    My office mate and I (I handle Epoch, he handles Unitree) have enjoyed comparing the merits/demerits of each over the last year. Comparison in our case is slightly slanted due to the fact that the Epoch has optical disk while the Unitree system has 3480 tape - so some observations have more to do with media advantages/disadvantages.

    
            Unitree
    
    +  Allows large files - can span volumes
    +  Allows you to enlarge the cache easily, allows very large 
    cache
    +- Unitree has replaced several UNIX utilities with their own 
    (FTP, NFS and the file system).  This allows certain features to 
    work but is generally slower and disallows access to the archive when 
    you are on the server itself - unless you NFS mount!
    +  Allows deleted files to be saved for a specified time
    +  Allows multiple copies of files to be kept
    +  Data is copied to archive soon after creation
    +  Unitree runs on several different platforms
    
  • Does not allow access to data until it is completely reloaded
  • Behaves poorly with small files (due to necessary overhead)
  • Unitree is licensed to several vendors, so versions differ
  • NFS access is so slow that we recommend it not be used for file transfer - only for ls and du, etc. Use FTP.
  • The Silicon Graphics version is still new and has some problems Epoch + Allows access to the data as soon as part of it is loaded + Company seems serious about reputation and support + The Epoch II is based on a SUN system, with few modifications + Data is copied to archive only when the cache space is needed + All native UNIX transfer methods work - NFS, FTP and RCP + Add on products greatly simplify backup and extend archiving features to other systems.
  • Deleted files are gone forever
  • Currently only available on SUN. This will change.
  • Cannot span volumes yet - limiting file size
  • Has the SUN limitation of 2 Gb per filesystem. This would be a bigger problem if you used it for a 3480 silo. {Note 2GB of Magnetic Disk limit, not the entire HSM store}
  • Behaves poorly with small files (due to necessary overhead)
  • Since inodes are kept on magnetic cache, you must take into account the maximum number of files you will ever need.
  • Since inodes are always on disk, certain disk operations can take forever since all inodes must be examined.
  • Enlarging a magnetic disk filesystem which has associated archive media requires you to offload all data and then reload it. If anyone has found another way, I would like to hear about it. {Others did offer some easier work-arounds}
  • In all fairness to Titan, they have been addressing any problems and it has been improving. Epoch too plans to address some of their shortcomings. We are looking forward to growing with both products.

    The likelihood that the various flavors of Unitree will standardize depends on what happens with Discos. My guess is that some features/enhancements will be filtered back to the base product released by Discos. Bye...

    (bodoh@dgg.cr.usgs.gov, 152.61.192.66, Tom Bodoh, USGS/EROS Data Center, Sioux Falls, SD)


    3.2. National Storage Lab {Brief}

    NSL is an industry consortium (American companies only) that has a version of Unitree, and is creating their own new High Performance Storage System.

    HPSS, among other features, supports striping of removable media, and full 64-bit files. Some of the work is being done at LLNL, where UniTree was originally developed. The HPSS page is at ORNL.

    (rdv,95/1/12)


    3.3. HIARC {New}

    HIARC HSM runs on Solaris 2.4 and above. Slides in at the vnode layer. Supports 4mm, 8mm, 3480, DLT, VHS, D-1 and D-2 tape drives, and appropriate robotics (I don't have a specific list). Removable media formats are standard (_which_ standard, I don't know). Pricing from $4k to $25k is reasonable for the functionality. See www.hiarc.com. (rdv, 97/3/20)


    3.4. Epoch (also known as StorageTek's NearNet) {Brief}

    See also "Epoch vs Unitree" in Appendix


    3.5. Zetaco/NETstor {Brief}

    NETstor can be reached at netstor-sales@netstor.com

    NETstor, Inc. (formerly Zetaco, Inc.) is a leading provider of hierarchical online mass-storage systems for open systems. Primarily NFS accessable systems with magnetic disks and optical-disk libraries. They have marketing agreements with Digital Equipment Corp, and Hewlett-Packard.

    (stith@fnal.gov)

    Netstor was bought by Cheyenne, and is now sold by them (lily@access.digex.com, 10/95).


    3.6. R-Squared Infinity IFS 2 {Brief}

    Contact: Steve Wine, Manager, Mass Storage Products, R-Squared, 11211 East Arapahoe Rd, Englewood, CO 80112, 303/799-9292 or FAX 303/799-9297


    3.7. AMASS

    From Advanced Archival Products. Supports a huge range of devices, autochangers, and operating systems. Block-based movement of data between the hard disk cache and tape or optical tertiary storage. Systems run from a few gigabytes up to at least 12 TB, with prices dependent on capacity. New versions allow multiple cache disks. Slips right in to the VFS layer and looks like a normal Unix file system, with the plusses and minuses that entails. No file versioning or multiple copies yet. File creation is an Achilles' heel on performance. Since it's block based, files can be larger than a piece of media. Separate product DataMgr will migrate files from client machines to the AMASS server automatically (with FS changes, of course).

    AMASS is now owned by EMASS, and you can find info at www.emass.com. (rdv, 1996/3/27)


    3.8. Tracer XFS {None}


    3.9. Metior

    Metior (pronounced like meteor) is targetting an incredibly broad market, from laptops with removable media through supercomputers, with prices from $650(!) to $118K. They handle multiple coordinated copies, so off-site backup can be automatic. Can do migration for client machines (with appropriate software licenses and changes to the file system). The hierarchy seems to be extremely flexible, variable on a per-user or per-group basis. Machines without client licenses can mount the Metior FS using NFS. Runs on Suns, SGI, and HP 9000/700. ANT is new, and they've only got a handful of customers so far, but it looks _very_ interesting.

    (info from habbott@csn.org, written by rdv, so it's my fault if it's not accurate) (rdv,94/7/7)

    More information available on the WWW FAQ version. Click here to see an email conversation I had with Hal. Also see them at anthill.com.

    Automated Network Technologies
    3333 South Bannock Street, Suite 945
    Englewood, CO 80110   USA
    Phone 303.789.2506
    FAX 303.789.2438
    Email hal@anthill.com
    


    3.10. NAStore {Brief}

    NAStore is a Unix migrating file system developed by the Numerical Aerodynamic Simulation program at NASA Ames. It is available through NASA's software distribution agency, COSMIC. It currently runs only on Convex with 34x0 cartridges and Storage Tek robots. Looks like a local file system to users of the Convex. Available with source.

    Info on NAStore can be found on the web at chuck.nas.nasa.gov

            COSMIC's address is :
            University of Georgia
            382 East Broad Street
            Athens, Georgia, 30602-4272, US
            011-706-542-3265
            service@cossack.cosmic.uga.edu
    

    For more information on NAStore, contact John Lekashman, lekash@nas.nasa.gov. (info from Bill Ross, bross@nas.nasa.gov, 94/9/15)


    3.11. DMF {Brief}

    Cray Research's Data Migration Facility. The grandaddy of Unix HSM systems. You can find info on DMF at www.cray.com, or call +1 612 683 3897 or email crayinfo@cray.com. It's reportedly running on more than 200 systems, and development is continuing. Large users are in the hundreds of TB, with millions of files and >1TB/day through DMF.

    Information from: "Storage Management at Cray Research, Inc", Metcalfe, D.J. and Thompson. D. "Data Migration Facility Development Update", Lazatella, T.W. and Bannister, N. Cray User Group, Barcelona, 1996, in press.

    (Robert.Bell@dit.csiro.au, 1996/4/2)


    3.12. FileServ {Brief}

    From E-Systems. Works with the E-Systems ER-90 (D-2) tape drive and Odetics robots, as well as 3480 with the Storage Tek ACS 4400. Runs on Convexes (only?). Supports multiple copies of files. Retrieves only necessary info from tape to disk before completing request.

    Reportedly no longer available on Convex, in beta test on SGI (lily@access.digex.net, 10/95)

    Now owned by EMASS, info at www.emass.com.


    3.13. Cray Research's Open Storage Manager {Brief}

    They have some agreement with Legent Corporation. OSM runs on Sparc machines, including the Cray Superservers. Price ranges from $500 to $5,000, which is very cheap for HSM. However, it might only be capable of migrating among disks -- I don't see any mention of autochangers. (rdv, 94/12/9)


    3.14. T-mass {None}


    3.15. HP OpenView OmniStorage

    Supports multiple types of tertiary media (optical, tape) though it seems to come originally from their work for their own MO jukeboxes. Supports multiple types of clients. (info from Herbert Volk , 1995/9/28)

    More info available at www.hp.com. Now a very broad storage management suite, covering lots of functionality for management. Supports MO, DLT and 8mm as media, though only a limited number of autochangers. (rdv, 98/1/16)


    3.16. Platinum NetArchive-HSM {Brief, New}

    Used to be ASC (Advanced Systems Concept) before being bought by Platinum. Runs on SunOS, HP, and Domain/OS. Supports numerous optical jukeboxes. See www.platinum.com. (rdv, 96/4)

    PLATINUM technology, inc.
    1815 South Meyers Road
    Oakbrook Terrace, IL 60181
    1-800-442-6861  -or-  708/620-5000
    e-mail: info@platinum.com
    


    3.17. Large Storage Configurations {Brief,New}

    www.lsci.com describes their Solaris-based HSM product. Only one computing platform, but a reasonably broad range of mid- to high-end peripherals and robotics supported, from little Exabyte autochangers to the IBM 3494 and STK silos. (rdv, 96/7/23)


    3.18. Unix HSM Vendor List

    This list is adapted from _Client/Server Today_, Dec. '94, with some of my own additions. All the phone numbers are USA (apologies to international readers for the 800 numbers, but they're all I've got). I don't know anything about some of these companies; I suspect some of them work with HSM from other vendors rather than their own packages.

    I've indicated on the list various reports of companies OEMing from each other; this is not out of disrespect for the work involved in OEMing/supporting or porting such complex software, but an attempt to divide the HSM vendors into "families" with similar capabilities (occassionally on very disparate platforms).

    Vendor					Product		Contact
    ------					-------		-------
    Advanced Archival Products		AMASS		(303)792-9700 *
    Advanced Software Concepts (ASC)			(619)737-9544
    Alphatronix				ASC		(919)544-0001
    Artecon					ASC		(619)931-5500
    AT&T CommVault				DataMigrator	(908)935-8000
    Automated Network Technologies (ANT)	Metior		(303)789-2506 *
    Computer Associates International			(800)225-5244
    Computer Upgrade					(808)874-8807
    Convex Computer				UniTree		(214)497-3085 *
    COSMIC (NAStore)					(706)542-3265 +
    Cray Research				DMF		(800)BUY-CRAY *
    Digital Equipment (DEC)			NETstor		(800)344-4825
    Dorotech						(703)478-2260
    Epoch Systems						(508)836-4300 *
    E-Systems				FileServ		      ?*
    File Tek				Storage Machine	(301)251-0600
    Fujitsu Computer Products of America	OSM		(408)432-6333
    Hewlett-Packard			OmniStorage* ,NETstor	(800)637-7740x8509
    HIARC							(714)253-6990
    IBM					UniTree		(800)225-5426
    Introl							(612)788-9391
    Large Software Configurations (LSC)			(612)482-4535 *
    Legent					$OSM		(703)708-3000
    National Storage Lab (NSL) 		HPSS			      +*
    NETstor (Cheyenne)			$NETstor	(612)890-9367
    (OpenVision				UniTree		(510)426-6400 *)
    Platinum				NetArchive HSM  (708)620-5000 *
    Qstar Technologies					(301)762-9800
    Raxco							(301)258-2620
    Software Partners/32					(508)887-6409
    Storage Technology (STK, StorageTek)			(303)673-5151
    T-mass						     		      ?
    Tracer XFS							      ?
    UniTree Software			UniTree		(510)833-9344 *
    
    * = Info elsewhere in FAQ
    + = not commercial product
    ? = no contact info
    $ = original developer (no mark indicates OEM)
    


    3.19. Mainframe

    IBM also has HSM for MVS, called, imaginatively, HSM.

    There is the storage home page. www.storage.ibm.com I have also found references to System Managed Storage SMS and HSM and DFHSM (Data Facility Hierarchical Storage Manager) but could find no online information. There are probably manuals like DFHSM Version 2 Release 5.0, General Information manual (GH35-0092) if you are a real glutton for punishment and have a friend at ibm.

    So we have ADSM and DFHSM and DFSMS and probably others. But not much online information. Sorry.

    A little searching from the www.ibm.com might turn up something too.

    (Del Cecchi, , 1996/3/27)


    3.20. PC & PC Server Oriented Packages


    3.20.1. HP Optical Jukebox Storage Solution

    Netware 3.11 based, up to 10.4 Gigabytes, includes model 10LC optical jukebox which has one drive and 16 disks each with 650 MB formatted capacity. Hewlet-packard (Palo Alto, CA) 800/826-4111.


    3.20.2. Chili Pepper Software

    A company from Atlanta, GA named Chili Pepper Software (404-339-1812) and 3M have gotten together in some fashion to make HSM software for PCs using QIC. (rdv, 94/9/5)


    3.20.3. Cheyenne ARCserve

    Runs on Netware servers. Transparent to most clients, but has a neat feature: if you use a special TSR and DLL on client PCs, when it has to retrieve a file from secondary or tertiary storage, it can give you an estimated retrieval time and the option to abort. (516)484-5110, (800)243-9462. (rdv,95/02/14)


    3.21. DATMAN {Brief}

    Simple HSM for 4mm tape drives under MS-DOS. A limited freeware version is available.

    More info at www.datman.com. Voice: 708-369-7112 Fax: 708-369-7113 (Kan Yabumoto, yabumoto@datman.com, Nov. 1995)


    3.22. Windows NT

    Try:

    Avail Systems
    4760 Walnut St
    Boulder, CO  80301
    voice: +1.303.444.4018
      fax: +1.303.546.4219
    

    dave_skinner@intellistor.com (Dave Skinner) (95/2/12)

    Avail's product, NetSpace HSM, has been selected by Microsoft to be incorporated into future versions of NT, and also provides a link between NetWare and IBM's ADSM. NetSpace also runs on Novell NetWare systems. See www.avail.com ("Wight, Risa" , 95/10/17)


    3.23. Other Non-Unix HSM

    DEC's old Tops-20 OS supported offline files, and would generate an automatic request to the operator to mount a tape when the user accessed the file. When you listed a directory, it would show you which files were online and which off.

    DEC's OpenVMS has some sort of support for this now. VMS 6.1 supports "shelved" files.

    There is also the product Virtual Branches, from Acorn Software, which does HSM for MO and CD-ROM for OpenVMS.

    
    Acorn Software, Inc.
    267 Cox St.
    Hudson, MA  01749
    voice:  (508)568-1618
    fax:    (508)562-1133
    Internet:  info@acornsw.com
    
    


    3.24. Tapes as Disks {Brief, New}

    There are several packages around (mostly for PCs) that will let you use a tape drive like a disk drive. Of course, it's _very_ slow unless it uses some disk-based information as well.

    See www.tapedisk.com for one such product. (rdv, 96/11/4)


    My Home Page at Caltech

    email me at rdv@isi.edu

    Copyright 1996 Rod Van Meter