White Paper: Universal Disk Format
PURPOSE
This white paper introduces Universal Disk Format (UDF), describes how it came into existence, and discusses advances in its evolution.
LOGICAL FILE FORMATS
Information in the form of digitally encoded data must be kept in specific, pre-defined locations on a storage medium in order for it to be read by a computer. File locations differ depending upon the operating system being used because each operating system in existence today uses its own proprietary logical file format.
Since each operating system has its own method of filing or finding information on media, removable media is incapable of being easily ported between two computers running different operating systems. This cross-platform limitation prohibits data interchange; information cannot be shared easily, especially if the two computers are not connected to a network.
In order to address the interoperability limitations of modern file systems, standards have been created and defined.
HIGH SIERRA AND THE ISO 9660 STANDARD
CD-ROM is the world's standard media containing software applications, periodicals, legal databases, medical databases, government regulations, catalogs, reference materials, knowledge bases and business critical information.
Early in the evolution of CD-ROM technology a committee of engineers known as the High Sierra Group authored a standard to define the requirements for reading CDs on a variety of computing platforms. With some minor modifications, this standard was quickly adopted by the International Standards Organization (ISO). Know as ISO 9660, it defined the way data was arranged in the sectors and tracks of a CD.
Today, ISO 9660 is an industry-wide standard ensuring that users can play CDs on virtually any CD-ROM drive, regardless of the specific platform. ISO 9660 is largely responsible for the success of the CD-ROM worldwide.
Unfortunately, the ISO 9660 standard is limited to mainly CD-ROM and CD-R technology. Therefore, its usefulness does not easily extend to other media technologies, including a wide variety of rewritable technologies.
WHAT IS UDF AND WHY IS IT IMPORTANT?
Each operating system in existence today uses a proprietary file system format preventing removable writable media from being portable across operating systems. For example, writable media created using OS/2's native HPFS file system format cannot be read or modified with Windows NT's native file system. In order to address this limitation of modern file systems, another new file system has been created and defined.
The Universal Disk Format (UDF) is a universal vendor-independent file system designed for data interchange and portability allowing an operating system to access (read/write/modify) data that was created by another operating system. UDF is a subset of - and is fully compliant with - an International Standards Organization (ISO) interchange standard for non-sequential recording (NSR) of data, known as ISO 13346. UDF is the subset of ISO 13346 as defined by the Optical Storage Technology Association (OSTA), a non-profit trade association actively promoting the use of writable optical technologies and products for storage of computer data and images.
UDF can be thought of as the next generation of ISO 9660, the read-only file system standard that enabled CD-ROM media to be read on all operating systems. ISO 9660 was largely responsible for the success of the CD-ROM in the home and in commercial and governmental organizations worldwide. UDF goes beyond ISO 9660, however, in that it is a writable file system standard enabling files on writable storage devices and media to be read/written/modified by a wide variety of operating systems.
The primary goals of UDF are to maximize robustness of data interchange, assure uniformity of data, support platform-unique file information, and minimize the cost and complexity of implementing ISO 13346. UDF defines information stored by a specific operating system (OS), information stored by all OSs, and how to process OS-specific information. It is, therefore, ideal for any application or usage where file interchange among different operating systems is required.
UDF is an important standard for data storage. It offers data interchange between operating systems, data interchange between vendor-unique software applications, support for massive volumes of data, and offers the only ISO standard file system for WORM and DVD media. UDF is anticipated to promote the rapid adoption and disbursement of many new media technologies on multiple platforms.
To date, the UDF file system standard has been specified for DOS/Windows 3.xx, OS/2, OS/400, Macintosh, Unix (all versions and flavors), Windows 9x, and Windows NT operating systems.
UDF Volume limits
Volume Size - unlimited
Logical Sector Size - unlimited (must be multiple of 512)
Logical Sectors per Volume - 232-1
Partition Size - 232-1 logical sectors
Logical Volume Size - 232-1 partitions * Maximum Partition Size
Volume Set Size - 216-1 Volumes
UDF Directory Limits
Directory Size - 264-1 bytes
Sub-directories per Directory - 216-1 sub-directories
Directory Name - 256 bytes
UDF File Limits
Logical Block Size - 232-1 partitions * Maximum Partition Size
File Extent Size - 230-1 bytes
File Size - 264-1 bytes
File Name - 256 bytes
THE EVOLUTION OF UDF
To understand how UDF came into existence, it is meaningful to describe how the file system standards advanced.
Promotion of the file system standards began in October 1991 with the American National Standards Institute (ANSI) sub-committee called X3B11.1. The primary objective of this sub-committee was to develop a standard file system for WORM and rewritable media. Micro Design International (MDI) played an active role in the work being done by the sub-committee and participated in monthly meetings.
During the development process, the sub-committee invited participation from the European Computer Manufacturers Association (ECMA) standards group in Europe. Through ECMA's involvement in the standard, the X3B11.1 sub-committee determined the best way to take the draft NSR standard forward on the path to becoming an ISO standard was to first have it become an ECMA standard. At a meeting in June 1992, the ECMA organization voted on and approved the finalized NSR draft document as the standard ECMA 167.
ECMA 167 was submitted to the ISO Fast Track early in 1993. ISO Fast Track takes an existing National Standard and submits it to a 6-month review and vote process. During this review process the voting member countries of ISO review the document to determine if it should become an ISO standard. On November 8, 1993 in Sarasota, Florida the ISO SWG committee met to review the results of the Fast Track vote and determined the final content of the ISO 13346 standard. MDI headed the U.S. delegation to this meeting, voting yes.
The yes vote for the U.S. contained comments regarding technical changes that needed to be made to ECMA 167 before becoming the standard ISO 13346. The X3B11.1 sub-committee proposed the list of technical changes submitted by the U.S. Several of the technical changes were a result of deficiencies discovered in ECMA 167 during their early implementation of the file system standard. As a result of these changes and others, minor technical changes were made and the ISO 13346 standard was approved. ECMA republished ECMA 167 as ECMA 167 Second Edition, which is technically identical to ISO 13346.
ISO 13346 was a complex standard and needed a set of guidelines written if there were to be true data interchange among various implementations. In the fall of 1993 a draft document that addressed how to implement ISO 13346 for DOS, OS/2, and UNIX was submitted to the newly formed OSTA Technical Committee. This initial draft evolved into today's UDF standard. MDI has played a leading role in the development of the UDF specification and the engineering effort to implement the UDF specification.
UDF REVISION HISTORY
UDF 1.00 - Original release
UDF 1.01 - DVD appendix added
UDF 1.02 - Final changes for DVD added
UDF 1.50 - Support for CD-R and CD-RW packet writing, support for Windows 95 and Windows NT (limited)
UDF 2.00 - Support for named streams, support for access control lists, and support for power calibration
UDF 2.01 - Support for real time files, support for OS/400
USING THE UDF FILE SYSTEM ON REMOVABLE MEDIA
UDF is a versatile file system supporting a wide variety of media technologies, such as CD, DVD, MO, WORM and magnetic disk. Because UDF is a writable - in addition to read-only - standard, UDF has been specified to work on a variety of new media technologies as shown below in Chart 1.
From a file system perspective, many new media technologies are related to existing technologies. All removable media fits into a relational hierarchy, shown in Figure 2 below. The following section discusses each part of this media hierarchy in turn.
Read-only (mastered) media
Media that is mastered in a media replication facility - such as CD-ROM or DVD-ROM - is read-only to the user. Thus, the file system driver functions as read-only, avoiding deletes, renames, space allocation, extending and truncating files, creation and removal of directories, and so on.
Writable media can be divided into two categories: sequentially written and non-sequentially written
Sequentially written media includes CD-R and DVD-R, which are similar on all levels with the exception of capacity. Sequential write generally refers to mastering, disk-at-once, or track-at-once recording, which lacks drag-and-drop ease-of-use and does not treat the drive or media as a drive-letter. Packet written CD-R or DVD-R, however, allows small amounts of data to be recorded with little overhead so it is visible as a drive letter.
With packet writing, more space is consumed on the media than the driver requests to write. This overrun must be adjusted for on subsequent write requests, as the file system driver must know the exact location where information is stored. Another UDF structure, the Virtual Allocation Table (VAT) allows remapping of blocks, which allows the file system driver to rewrite only changed information on updates. The VAT keeps track of the last known good state (previous image of the UDF file system that was properly closed) allowing the ability to flashback to the last recorded sector, eliminating the requirement for a separate check and repair utility. Non-sequentially written media includes MO, WORM, CD-RW, DVD+RW, DVD-RAM, and magnetic disk. Non-sequentially written media demands a higher level of complexity because free space management is required of the file system driver.
First, the UDF file system driver must keep track of unallocated space (space that is ready for allocation) and freed space (space that may require preparation before allocation). UDF must be able to hand out space and take it back.
Second, the UDF file system driver must optimize for meta-data changes. In other words, when files and directories are modified, UDF must determine how best to reuse existing allocated meta-data space. For example, when a directory entry is created, rather than simply adding a directory record to the end of the directory file (sequentially written approach) UDF must search for and reuse an existing directory record of the correct size that is no longer used.
Finally, the UDF file system driver must handle syncing vs. performance issues. Cached data can reflect data that resides anywhere on the medium. In the event of system failure, it cannot be known what did not get written to the media so an inspection and repair of the entire volume (FSCK) must occur before the volume can be accessed again. Depending upon the amount of time required to repair a damaged volume, a file system driver must be concerned with the amount of data that is kept cached since purging cached data may dramatically affect performance (not to mention thrashing in a jukebox of spanned volumes). Thus, the optimal sync vs. performance criteria must be established per platform and per product
Non-sequentially written media requires a check and repair utility to guarantee correct file system behavior in the event of a power outage or system crash while writing is taking place or information is cached in RAM.
Non-sequentially written media can be divided into write-once read-many (WORM) and rewritable
WORM media allows writes to any sector(s) on the disk in any order, but may write to any particular sector only once. WORM media must employ different directory meta-data strategies (strategy 4096) and allocation (space table) strategies defined in UDF to account for this behavior. Beyond this WORM media requires a repair utility, but the constraints are different due to its write-once behavior and the history inherent in its write strategy.
Rewritable media allows writes to any sector(s) on the disk in any order. Writes may occur to any sector again and again, similar to magnetic disk.
Rewritable media can be divided further into two categories depending upon the sector handling of the drives: defective sector handling and non-defective sector handling
Drives handling defective sectors require no involvement of a file system driver to maintain defects. Drive technologies that do handle defects on their own are MO, DVD-RAM and magnetic disk.
Drives that do not handle defective sectors require the UDF file system driver to handle this through software using UDF structures on disc. Therefore, all reads and writes must go through a remapping table known as the sparing table. This table of defective sectors must be maintained by the UDF file system and certain aspects of packet writing must be handled. Examples would include partial packet failures, caching and packing together file data from various files into a single packet, and so forth. Drive technology that does not handle defective sectors includes CD-RW and DVD+RW.
Though read-only and write-once mediums are important as storage mediums, rewritable UDF functionality is strategically more important due to usage patterns and historical adoption of new media technologies in the marketplace. Market demands show that users desire "floppy-like" drive letter access (save as, copy, move, etc.) to information. Additionally, most users prefer to have the option to delete some files as they age or become unnecessary. This usage pattern is not consistent with write-once technology due to the fact that data cannot be altered once written to the media.
It is expected that the market will move from CD to DVD over time due its high capacity and adherence to the UDF file system standard for both read-only and rewritable technologies. The currently write-once DVD technology (DVD-R) is very expensive and it is not anticipated to be cost-effective until such time as rewritable DVD technology becomes cost effective, increases in popularity, and matures into the mainstream (replacing VCR in the home). Micro Design International, Inc.