| Main | Site Index | Download | Disclaimer | Privacy |


 

 

 

SUPPLEMENTARY NOTE

For Windows Storage and File System Story

 

 

 

What do we have in this Module?

  1. Windows File Systems

  2. File System Components

  3. Volume

  4. Directory

  5. File

  6. Windows Supported File Systems

  7. Comparisons of Some File System Features and Limits

  8. File Security and Access Rights

  9. A Backup Story

  10. Input and Output (I/O)

  11. Synchronous and Asynchronous I/O

  12. File Caching

  13. Alertable I/O

  14. I/O Completion Ports

  15. Network I/O Concepts

  16. Description of a Network I/O Operation

  17. Microsoft SMB Protocol and CIFS Protocol

  18. Opportunistic Locks

  19. Processes and Threads

 

 

 

 

 

 

 

 

 

 

This note should be read together with Windows File System tutorial and that follows. A file management in general can be found in File Management ppt slide.

 

Windows File Systems

  • A file system enables applications to store and retrieve files on storage devices.  Files are placed in a hierarchical structure.

  • The file system specifies naming conventions for files and the format for specifying the path to a file in the tree structure.

  • Each file system consists of one or more drivers and dynamic-link libraries that define the data formats and features of the file system.  File systems can exist on the following storage devices:

  1. Diskettes.

  2. Hard disks.

  3. Jukeboxes.

  4. Removable optical disks – CDR/RW, DVD R/RW.

  5. Tape backup units.

File System Components

  • All file systems supported by Windows have the following storage components:

Volume

  • A volume is a collection of directories and files.  The highest level of organization in the NTFS file system is the volume.

  • A volume is an area on a storage device that is managed by the file system as a discrete logical storage unit.

  • A partition contains at least one volume, and a volume can exist on one or more partitions.  A volume that contains data that exists on one partition is called a simple volume, and a volume that contains data that exists on more than one partition is called a multipartition volume.

Directory

  • A Directory is a hierarchical collection of directories and files.  A directory is the file system component that contains one or more files or directories.

  • The only constraint on the number of files that can be contained in a single directory is the physical size of the disk the directory is located on.

  • A directory that contains one or more directories is the parent of the contained directory or directories, and each contained directory is a child of the parent directory.  The hierarchical structure of directories is referred to as a directory tree.

  • NTFS implements the logical link between a directory and the files it contains as a directory entry table.

  • When a file is moved into a directory, an entry is created in the table for the moved file and the name of the file is placed in the entry.

  • When a file contained in a directory is deleted, the name and entry corresponding to the deleted file is also deleted from the table.  More than one entry for a single file can exist in a directory entry table.

  • If an additional entry is created in the table for a file, that entry is referred to as a hard link to that file.  There is no limit to the number of hard links that can be created for a single file.  Directories can also contain junctions and reparse points.

File

Windows Supported File Systems

  1. NTFS

  2. FAT32

  3. FAT16 and FAT12

  4. UDF

  1. File names can be 255 characters long.

  2. File names can be uppercase and lowercase.

  3. The maximum path length is 1023 characters.

  1. CDFS

  1. Directory and file names must be fewer than 32 characters long.

  2. Directory trees can be no more than eight levels deep.

 

Comparisons of Some File System Features and Limits

Feature

NTFS

FAT32

FAT16

MS-DOS compatibility

No

No

Yes

Disk quotas

Yes

No

No

File compression

Yes

No

No

Limit

NTFS

FAT32

FAT16

File size

264 - 1 bytes

232 - 1 bytes

232 - 1 bytes

Minimum cluster size

512 bytes

512 bytes

512 bytes

Maximum cluster size

64 KB

64 KB

64 KB

Minimum volume size

1 MB

2 GB

2,091,520 bytes

Maximum volume size

232 allocation units

4,177,198 clusters

4 GB

Windows Me/98/95:   2 GB

Files per volume

232 - 1

228

216

Files or directories per directory

Unlimited

216 - 2

216 - 2

  1. The maximum amount of storage space.

  2. The maximum number of disk drives per server.

  3. The maximum number of open local files.

  4. The maximum number of simultaneous file locks.

File Security and Access Rights

Access right

Description

FILE_ADD_FILE

For a directory, the right to create a file in the directory.

FILE_ADD_SUBDIRECTORY

For a directory, the right to create a subdirectory.

FILE_ALL_ACCESS

All possible access rights for a file.

FILE_APPEND_DATA

For a file object, the right to append data to the file. For a directory object, the right to create a subdirectory.

FILE_CREATE_PIPE_INSTANCE

For a named pipe, the right to create a pipe.

FILE_DELETE_CHILD

For a directory, the right to delete a directory and all the files it contains, including read-only files.

FILE_EXECUTE

For a native code file, the right to execute the file. This access right given to scripts may cause the script to be executable, depending on the script interpreter.

FILE_LIST_DIRECTORY

For a directory, the right to list the contents of the directory.

FILE_READ_ATTRIBUTES

The right to read file attributes.

FILE_READ_DATA

For a file object, the right to read the corresponding file data. For a directory object, the right to read the corresponding directory data.

FILE_READ_EA

The right to read extended file attributes.

FILE_TRAVERSE

For a directory, the right to traverse the directory. By default, users are assigned the BYPASS_TRAVERSE_CHECKING privilege, which ignores the FILE_TRAVERSE access right. See the remarks later in this section for more information.

FILE_WRITE_ATTRIBUTES

The right to write file attributes.

FILE_WRITE_DATA

For a file object, the right to write data to the file. For a directory object, the right to create a file in the directory.

FILE_WRITE_EA

The right to write extended file attributes.

STANDARD_RIGHTS_READ

Includes READ_CONTROL, which is the right to read the information in the file or directory object's security descriptor. This does not include the information in the SACL.

STANDARD_RIGHTS_WRITE

Includes WRITE_CONTROL, which is the right to write to the directory object's security descriptor. This does not include the information in the SACL.

SYNCHRONIZE

The right to specify a file handle in one of the wait functions. However, for asynchronous file I/O operations, you should wait on the event handle in an OVERLAPPED structure rather than using the file handle for synchronization.

 

Access right

Description

GENERIC_EXECUTE

FILE_READ_ATTRIBUTES

STANDARD_RIGHTS_EXECUTE

SYNCHRONIZE

GENERIC_READ

FILE_READ_ATTRIBUTES

FILE_READ_DATA

FILE_READ_EA

STANDARD_RIGHTS_READ

SYNCHRONIZE

GENERIC_WRITE

FILE_APPEND_DATA

FILE_WRITE_ATTRIBUTES

FILE_WRITE_DATA

FILE_WRITE_EA

STANDARD_RIGHTS_WRITE

SYNCHRONIZE

A Backup Story

  • A backup application would not be able to complete its job of backing up your file if the access restrictions you have placed on your file or directory does not allow the application's user-mode process to read it.

  • Backup applications must be able to override the security settings of file and directory objects to ensure a complete backup.

  • Similarly, if a backup application attempts to write a backup copy of your file over the disk-resident copy, and you explicitly deny write privileges to the backup application process, the restore operation cannot complete.  In this case also, the backup application must be able to override the access control settings of your file.

  • The SE_BACKUP_NAME and SE_RESTORE_NAME access privileges were specifically created to provide this ability to backup applications.

  • If these privileges have been granted and enabled in the access token of the backup application process, it can then call CreateFile() to open your file or directory for backup, specifying the standard READ_CONTROL access right as the value of the dwDesiredAccess parameter.

  • However, to identify the calling process as a backup process, the call to CreateFile() must include the FILE_FLAG_BACKUP_SEMANTICS flag in the dwFlagsAndAttributes parameter.  For example, the full syntax of the function call is the following:

hFile = CreateFile(fileName,

            READ_CONTROL,

            0,

            NULL,

            OPEN_EXISTING,

            FILE_FLAG_BACKUP_SEMANTICS,

            NULL);

hFile = CreateFile(fileName,

            WRITE_OWNER|WRITE_DAC,

            0,

            NULL,

            CREATE_ALWAYS,

            FILE_FLAG_BACKUP_SEMANTICS,

            NULL);

Input and Output (I/O)

Synchronous and Asynchronous I/O

  1. Do not de-allocate the OVERLAPPED structure or the data buffer until all asynchronous I/O operations to the file object have been completed.  If it is de-allocated prematurely, ReadFile() or WriteFile() may incorrectly report that the I/O operation is complete.

  2. If you declare your pointer to the OVERLAPPED structure as a local variable, do not exit the local function until all asynchronous I/O operations to the file object have been completed. If the local function is exited prematurely, the OVERLAPPED structure will go out of scope and it will be inaccessible to any ReadFile() or WriteFile() functions it encounters outside of that function.

File Caching

Alertable I/O

  1. A thread initiates an asynchronous read request by calling ReadFile() with a pointer to a callback function.

  2. The thread initiates an asynchronous write request by calling WriteFile() with a pointer to a callback function.

  3. The thread calls a function that fetches a row of data from a remote database server.

  1. SleepEx()

  2. WaitForSingleObjectEx()

  3. WaitForMultipleObjectsEx()

  4. SignalObjectAndWait()

  5. MsgWaitForMultipleObjectsEx()

 
  1. The kernel checks the thread's APC queue.  If the queue contains callback function pointers, the kernel removes the pointer from the queue and sends it to the thread.

  2. The thread executes the callback function.

  3. Steps 1 and 2 are repeated for each pointer remaining in the queue.

  4. When the queue is empty, the thread returns from the function that placed it in an alertable state.

  1. The kernel object that is being waited on becomes signaled.

  2. A callback function pointer is placed in the APC queue.

I/O Completion Ports

Network I/O Concepts

Description of a Network I/O Operation

  1. The I/O request is intercepted by a network redirector, also referred to simply as a redirector, on the local computer.  This is depicted in the preceding figure by the solid arrow between the application and the client redirector.

  2. The redirector constructs a data packet containing all of the information about the request, and sends it to the server where the file is located.  This is depicted in the preceding figure by the solid arrow between the client redirector and the server redirector.

  3. The redirector on the server receives the packet from the client, authenticates the access to the file required by the I/O request, and, if authenticated, executes the request on behalf of the client.  If not, it returns an error code to the redirector on the client.  This is depicted in the preceding figure by the curved solid arrow between the server redirector and the file.

  4. When the request has been executed, the redirector on the server sends any data resulting from the I/O request to the redirector on the client along with a success notification.  This is depicted in the preceding figure by the dotted arrow between the server and the client redirector.

  5. The redirector on the client receives the packet from the server and passes the data in the packet to the application along with a success notification.  This is depicted in the preceding figure by the dotted arrow between the client redirector and the application.

Microsoft SMB Protocol and CIFS Protocol

  1. Determining other Microsoft SMB Protocol servers on the network, or network browsing.

  2. Printing over a network.

  3. File, directory, and share access authentication.

  4. File and record locking.

  5. File and directory change notification.

  6. Dialect negotiation.

  7. Extended file attribute handling.

  8. Unicode support.

  9. Opportunistic locks.

  1. Session control packets.  Establishes and discontinues a connection to shared server resources.

  2. File access packets.  Accesses and manipulates files and directories on the remote server.

  3. General message packets.  Sends data to print queues, mailslots, and named pipes, and provides data about the status of print queues.

Opportunistic Locks

Processes and Threads

 

 

---------------------------------------End of story---------------------------------

 

 

 

 

 

 

 

 

 

 

 

| Main | Site Index | Download | Disclaimer | Privacy |