3. Hierarchical Storage Management

HSM systems transparently migrate files from disk to optical disk and/or magnetic tape, usually robotically accessible. Then when files are accessed by a user, they transparently move them back to disk.

Watch for maximum file size limitations, sometimes limited by the size of the media used, sometimes by the server's OS, and sometimes neither.

Some offer integrated backup. Some will manage multiple copies of files for data reliability.

Some offer integrated migration from other systems (ie, file servers and/or workstations) to the central location disks, then to the central location robotics. This generally requires changes to the on-disk file system format on the migration clients.

An item to watch for is that the file management may be exactly like Unix -- that is, all files appear to be online, and once they're deleted, they're gone forever, even though the data may still be on tape.

All of the subsections here are Unix-compatible (in various flavors) unless indicated otherwise.

Additional Information:

See also _DEC Professional_, February 1993, Page 40 and _Client/Server Today_, Dec. '94, p. 60.

The System-Managed Storage Guide by Howard W. Miller, $225 for first copy, $75 for additional copies for same company available from The Information Technology Institute, 136 Orchard Street, Byfield, Massachusetts, 01922-1605.

(stith@fnal.gov)

Thomas Woodrow did an evaluation of NAStore, FileServ, DMF and Unitree in 1993. It can be obtained through www.nas.nasa.gov or the Proc. 3rd NASA Goddard Conference on Mass Storage Systems and Technologies, Oct. 1993, pp. 187--216. Somewhat dated now but excellent methodology for comparing HSMs.

The uncle of UNIX HSMs. Developed primarily at Lawrence Livermore National Laboratories. Commercialized by a company called DISCOS, then sold to OpenVision. UniTree was sold to UniTree Software in December, 1994. See www.unitree.com.

Many versions exist on different hardware platforms, including a National Storage Lab (NSL) UniTree commercialized by IBM - Fed Systems. It's also available on SGI, Convex, and Amdahl hardware, at least.

See also "Epoch vs Unitree" below

For Convex, try

Jim Wilson
214-497-3085
jrwilson@convex.com
Business Development
Data Management Applications
Convex Computer Corporation

For most other platforms, call Open Vision at (800)223-OPEN or (510)426-6400.

New info:

The latest release of UniTree, V1.9.1, has the following changes:
 Available directly from UniTree Software Inc.
 Support DEC, HP-UX, SGI, Sun
 GUI(Tcl/Tk) tools for installation and administration
 New name database structure
 Common Message Logger
 Parallel Migration and Staging
 Multiple Storage Hierarchy (Optical/Tape)
 FTP performance improvements (Read/Write 20MBs/16MBs)*
 NFS performance improvements (Read/Write 3.5MBs/2MBs)*
 Rule-based dynamic migration
 Support for new robots (e.g., STK 97xx)
 Support >2GB disk partitions on Sun
 64K File Families
 Configurable media and drive types
 Departmental File Server Configuration
 Compatible with most backup software (Legato, CAM, SMArch)

Demo copy available for download from web site: www.unitree.com

New resellers in Asia, Europe, Australia


* Measured on a dual cpu Sun Ultra3000 with 256MB and 10 disks
-- 
Francis Kim				Phone: (510) 833-3460
Director of Sales and Marketing		FAX:   (510) 833-9345
Unitree Software, Inc.			e-mail: francis@unitree.com
11875 Dublin Blvd. Suite A200E	        WWW: http://www.unitree.com
Dublin, Ca. 94568

(Note: this evaluation is old, and should be taken with a grain of salt. rdv, 3/96)

(6/93) We just bought both last year. We bought an Epoch I with the 20 GB EO and 327 GB worm. We will be upgrading it to an Epoch II soon. We also bought Unitree from Titan to run on a Silicon Graphics server and hook up to the STK 3480 silo. We hope to add more silos eventually.

Unitree is licensed based on storage capacity while Epoch is not. There may be an exception to this - STK just began reselling Epoch as the front end for their silos and I'm not sure how they handle licensing.

My office mate and I (I handle Epoch, he handles Unitree) have enjoyed comparing the merits/demerits of each over the last year. Comparison in our case is slightly slanted due to the fact that the Epoch has optical disk while the Unitree system has 3480 tape - so some observations have more to do with media advantages/disadvantages.

        Unitree

+  Allows large files - can span volumes
+  Allows you to enlarge the cache easily, allows very large 
cache
+- Unitree has replaced several UNIX utilities with their own 
(FTP, NFS and the file system).  This allows certain features to 
work but is generally slower and disallows access to the archive when 
you are on the server itself - unless you NFS mount!
+  Allows deleted files to be saved for a specified time
+  Allows multiple copies of files to be kept
+  Data is copied to archive soon after creation
+  Unitree runs on several different platforms
  Does not allow access to data until it is completely 
reloaded
  Behaves poorly with small files (due to necessary overhead)
  Unitree is licensed to several vendors, so versions differ
  NFS access is so slow that we recommend it not be used for 
file transfer - only for ls and du, etc.  Use FTP.
  The Silicon Graphics version is still new and has some 
problems

        Epoch

+  Allows access to the data as soon as part of it is loaded
+  Company seems serious about reputation and support
+  The Epoch II is based on a SUN system, with few 
modifications
+  Data is copied to archive only when the cache space is 
needed
+  All native UNIX transfer methods work - NFS, FTP and RCP
+  Add on products greatly simplify backup and extend 
archiving features to other systems.
  Deleted files are gone forever
  Currently only available on SUN.  This will change.
  Cannot span volumes yet - limiting file size
  Has the SUN limitation of 2 Gb per filesystem.  This would 
be a bigger problem if you used it for a 3480 silo.
{Note 2GB of Magnetic Disk limit, not the entire HSM store}
  Behaves poorly with small files (due to necessary overhead)
  Since inodes are kept on magnetic cache, you must take 
into account the maximum number of files you will ever need.
  Since inodes are always on disk, certain disk operations 
can take forever since all inodes must be examined.
  Enlarging a magnetic disk filesystem which has associated 
archive media requires you to offload all data and then reload it.  
If anyone has found another way, I would like to hear about 
it.
                   {Others did offer some easier work-arounds}

In all fairness to Titan, they have been addressing any problems and it has been improving. Epoch too plans to address some of their shortcomings. We are looking forward to growing with both products.

The likelihood that the various flavors of Unitree will standardize depends on what happens with Discos. My guess is that some features/enhancements will be filtered back to the base product released by Discos. Bye...

(bodoh@dgg.cr.usgs.gov, 152.61.192.66, Tom Bodoh, USGS/EROS Data Center, Sioux Falls, SD)

NSL is an industry consortium (American companies only) that has a version of Unitree, and is creating their own new High Performance Storage System.

HPSS, among other features, supports striping of removable media, and full 64-bit files. Some of the work is being done at LLNL, where UniTree was originally developed. The HPSS page is at ORNL.

(rdv,95/1/12)

HIARC HSM runs on Solaris 2.4 and above. Slides in at the vnode layer. Supports 4mm, 8mm, 3480, DLT, VHS, D-1 and D-2 tape drives, and appropriate robotics (I don't have a specific list). Removable media formats are standard (_which_ standard, I don't know). Pricing from $4k to $25k is reasonable for the functionality. See www.hiarc.com. (rdv, 97/3/20)

NETstor can be reached at netstor-sales@netstor.com

NETstor, Inc. (formerly Zetaco, Inc.) is a leading provider of hierarchical online mass-storage systems for open systems. Primarily NFS accessable systems with magnetic disks and optical-disk libraries. They have marketing agreements with Digital Equipment Corp, and Hewlett-Packard.

(stith@fnal.gov)

Netstor was bought by Cheyenne, and is now sold by them (lily@access.digex.com, 10/95).

Contact: Steve Wine, Manager, Mass Storage Products, R-Squared, 11211 East Arapahoe Rd, Englewood, CO 80112, 303/799-9292 or FAX 303/799-9297

From Advanced Archival Products. Supports a huge range of devices, autochangers, and operating systems. Block-based movement of data between the hard disk cache and tape or optical tertiary storage. Systems run from a few gigabytes up to at least 12 TB, with prices dependent on capacity. New versions allow multiple cache disks. Slips right in to the VFS layer and looks like a normal Unix file system, with the plusses and minuses that entails. No file versioning or multiple copies yet. File creation is an Achilles' heel on performance. Since it's block based, files can be larger than a piece of media. Separate product DataMgr will migrate files from client machines to the AMASS server automatically (with FS changes, of course).

AMASS is now owned by EMASS, and you can find info at www.emass.com. (rdv, 1996/3/27)

Metior (pronounced like meteor) is targetting an incredibly broad market, from laptops with removable media through supercomputers, with prices from $650(!) to $118K. They handle multiple coordinated copies, so off-site backup can be automatic. Can do migration for client machines (with appropriate software licenses and changes to the file system). The hierarchy seems to be extremely flexible, variable on a per-user or per-group basis. Machines without client licenses can mount the Metior FS using NFS. Runs on Suns, SGI, and HP 9000/700. ANT is new, and they've only got a handful of customers so far, but it looks _very_ interesting.

(info from habbott@csn.org, written by rdv, so it's my fault if it's not accurate) (rdv,94/7/7)

More information available on the WWW FAQ version. Click here to see an email conversation I had with Hal. Also see them at anthill.com.

Automated Network Technologies
3333 South Bannock Street, Suite 945
Englewood, CO 80110   USA
Phone 303.789.2506
FAX 303.789.2438
Email hal@anthill.com

NAStore is a Unix migrating file system developed by the Numerical Aerodynamic Simulation program at NASA Ames. It is available through NASA's software distribution agency, COSMIC. It currently runs only on Convex with 34x0 cartridges and Storage Tek robots. Looks like a local file system to users of the Convex. Available with source.

Info on NAStore can be found on the web at chuck.nas.nasa.gov

        COSMIC's address is :
        University of Georgia
        382 East Broad Street
        Athens, Georgia, 30602-4272, US
        011-706-542-3265
        service@cossack.cosmic.uga.edu

For more information on NAStore, contact John Lekashman, lekash@nas.nasa.gov. (info from Bill Ross, bross@nas.nasa.gov, 94/9/15)

Cray Research's Data Migration Facility. The grandaddy of Unix HSM systems. You can find info on DMF at www.cray.com, or call +1 612 683 3897 or email crayinfo@cray.com. It's reportedly running on more than 200 systems, and development is continuing. Large users are in the hundreds of TB, with millions of files and >1TB/day through DMF.

Information from: "Storage Management at Cray Research, Inc", Metcalfe, D.J. and Thompson. D. "Data Migration Facility Development Update", Lazatella, T.W. and Bannister, N. Cray User Group, Barcelona, 1996, in press.

(Robert.Bell@dit.csiro.au, 1996/4/2)

From E-Systems. Works with the E-Systems ER-90 (D-2) tape drive and Odetics robots, as well as 3480 with the Storage Tek ACS 4400. Runs on Convexes (only?). Supports multiple copies of files. Retrieves only necessary info from tape to disk before completing request.

Reportedly no longer available on Convex, in beta test on SGI (lily@access.digex.net, 10/95)

Now owned by EMASS, info at www.emass.com.

They have some agreement with Legent Corporation. OSM runs on Sparc machines, including the Cray Superservers. Price ranges from $500 to $5,000, which is very cheap for HSM. However, it might only be capable of migrating among disks -- I don't see any mention of autochangers. (rdv, 94/12/9)

Supports multiple types of tertiary media (optical, tape) though it seems to come originally from their work for their own MO jukeboxes. Supports multiple types of clients. (info from Herbert Volk , 1995/9/28)

More info available at www.hp.com. Now a very broad storage management suite, covering lots of functionality for management. Supports MO, DLT and 8mm as media, though only a limited number of autochangers. (rdv, 98/1/16)

Used to be ASC (Advanced Systems Concept) before being bought by Platinum. Runs on SunOS, HP, and Domain/OS. Supports numerous optical jukeboxes. See www.platinum.com. (rdv, 96/4)

PLATINUM technology, inc.
1815 South Meyers Road
Oakbrook Terrace, IL 60181
1-800-442-6861  -or-  708/620-5000
e-mail: info@platinum.com

www.lsci.com describes their Solaris-based HSM product. Only one computing platform, but a reasonably broad range of mid- to high-end peripherals and robotics supported, from little Exabyte autochangers to the IBM 3494 and STK silos. (rdv, 96/7/23)

This list is adapted from _Client/Server Today_, Dec. '94, with some of my own additions. All the phone numbers are USA (apologies to international readers for the 800 numbers, but they're all I've got). I don't know anything about some of these companies; I suspect some of them work with HSM from other vendors rather than their own packages.

I've indicated on the list various reports of companies OEMing from each other; this is not out of disrespect for the work involved in OEMing/supporting or porting such complex software, but an attempt to divide the HSM vendors into "families" with similar capabilities (occassionally on very disparate platforms).

Vendor					Product		Contact
------					-------		-------
Advanced Archival Products		AMASS		(303)792-9700 *
Advanced Software Concepts (ASC)			(619)737-9544
Alphatronix				ASC		(919)544-0001
Artecon					ASC		(619)931-5500
AT&T CommVault				DataMigrator	(908)935-8000
Automated Network Technologies (ANT)	Metior		(303)789-2506 *
Computer Associates International			(800)225-5244
Computer Upgrade					(808)874-8807
Convex Computer				UniTree		(214)497-3085 *
COSMIC (NAStore)					(706)542-3265 +
Cray Research				DMF		(800)BUY-CRAY *
Digital Equipment (DEC)			NETstor		(800)344-4825
Dorotech						(703)478-2260
Epoch Systems						(508)836-4300 *
E-Systems				FileServ		      ?*
File Tek				Storage Machine	(301)251-0600
Fujitsu Computer Products of America	OSM		(408)432-6333
Hewlett-Packard			OmniStorage* ,NETstor	(800)637-7740x8509
HIARC							(714)253-6990
IBM					UniTree		(800)225-5426
Introl							(612)788-9391
Large Software Configurations (LSC)			(612)482-4535 *
Legent					$OSM		(703)708-3000
National Storage Lab (NSL) 		HPSS			      +*
NETstor (Cheyenne)			$NETstor	(612)890-9367
(OpenVision				UniTree		(510)426-6400 *)
Platinum				NetArchive HSM  (708)620-5000 *
Qstar Technologies					(301)762-9800
Raxco							(301)258-2620
Software Partners/32					(508)887-6409
Storage Technology (STK, StorageTek)			(303)673-5151
T-mass						     		      ?
Tracer XFS							      ?
UniTree Software			UniTree		(510)833-9344 *

* = Info elsewhere in FAQ
+ = not commercial product
? = no contact info
$ = original developer (no mark indicates OEM)

IBM also has HSM for MVS, called, imaginatively, HSM.

There is the storage home page. www.storage.ibm.com I have also found references to System Managed Storage SMS and HSM and DFHSM (Data Facility Hierarchical Storage Manager) but could find no online information. There are probably manuals like DFHSM Version 2 Release 5.0, General Information manual (GH35-0092) if you are a real glutton for punishment and have a friend at ibm.

So we have ADSM and DFHSM and DFSMS and probably others. But not much online information. Sorry.

A little searching from the www.ibm.com might turn up something too.

(Del Cecchi, , 1996/3/27)

A company from Atlanta, GA named Chili Pepper Software (404-339-1812) and 3M have gotten together in some fashion to make HSM software for PCs using QIC. (rdv, 94/9/5)

Runs on Netware servers. Transparent to most clients, but has a neat feature: if you use a special TSR and DLL on client PCs, when it has to retrieve a file from secondary or tertiary storage, it can give you an estimated retrieval time and the option to abort. (516)484-5110, (800)243-9462. (rdv,95/02/14)

Simple HSM for 4mm tape drives under MS-DOS. A limited freeware version is available.

More info at www.datman.com. Voice: 708-369-7112 Fax: 708-369-7113 (Kan Yabumoto, yabumoto@datman.com, Nov. 1995)

Try:

Avail Systems
4760 Walnut St
Boulder, CO  80301
voice: +1.303.444.4018
  fax: +1.303.546.4219

dave_skinner@intellistor.com (Dave Skinner) (95/2/12)

Avail's product, NetSpace HSM, has been selected by Microsoft to be incorporated into future versions of NT, and also provides a link between NetWare and IBM's ADSM. NetSpace also runs on Novell NetWare systems. See www.avail.com ("Wight, Risa" , 95/10/17)

DEC's old Tops-20 OS supported offline files, and would generate an automatic request to the operator to mount a tape when the user accessed the file. When you listed a directory, it would show you which files were online and which off.

DEC's OpenVMS has some sort of support for this now. VMS 6.1 supports "shelved" files.

There is also the product Virtual Branches, from Acorn Software, which does HSM for MO and CD-ROM for OpenVMS.

Acorn Software, Inc.
267 Cox St.
Hudson, MA  01749
voice:  (508)568-1618
fax:    (508)562-1133
Internet:  info@acornsw.com

There are several packages around (mostly for PCs) that will let you use a tape drive like a disk drive. Of course, it's _very_ slow unless it uses some disk-based information as well.

See www.tapedisk.com for one such product. (rdv, 96/11/4)

email me at rdv@isi.edu

Copyright 1996 Rod Van Meter