796 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3Written by: Neil Brown4Please see MAINTAINERS file for where to send questions.5 6Overlay Filesystem7==================8 9This document describes a prototype for a new approach to providing10overlay-filesystem functionality in Linux (sometimes referred to as11union-filesystems). An overlay-filesystem tries to present a12filesystem which is the result over overlaying one filesystem on top13of the other.14 15 16Overlay objects17---------------18 19The overlay filesystem approach is 'hybrid', because the objects that20appear in the filesystem do not always appear to belong to that filesystem.21In many cases, an object accessed in the union will be indistinguishable22from accessing the corresponding object from the original filesystem.23This is most obvious from the 'st_dev' field returned by stat(2).24 25While directories will report an st_dev from the overlay-filesystem,26non-directory objects may report an st_dev from the lower filesystem or27upper filesystem that is providing the object. Similarly st_ino will28only be unique when combined with st_dev, and both of these can change29over the lifetime of a non-directory object. Many applications and30tools ignore these values and will not be affected.31 32In the special case of all overlay layers on the same underlying33filesystem, all objects will report an st_dev from the overlay34filesystem and st_ino from the underlying filesystem. This will35make the overlay mount more compliant with filesystem scanners and36overlay objects will be distinguishable from the corresponding37objects in the original filesystem.38 39On 64bit systems, even if all overlay layers are not on the same40underlying filesystem, the same compliant behavior could be achieved41with the "xino" feature. The "xino" feature composes a unique object42identifier from the real object st_ino and an underlying fsid number.43The "xino" feature uses the high inode number bits for fsid, because the44underlying filesystems rarely use the high inode number bits. In case45the underlying inode number does overflow into the high xino bits, overlay46filesystem will fall back to the non xino behavior for that inode.47 48The "xino" feature can be enabled with the "-o xino=on" overlay mount option.49If all underlying filesystems support NFS file handles, the value of st_ino50for overlay filesystem objects is not only unique, but also persistent over51the lifetime of the filesystem. The "-o xino=auto" overlay mount option52enables the "xino" feature only if the persistent st_ino requirement is met.53 54The following table summarizes what can be expected in different overlay55configurations.56 57Inode properties58````````````````59 60+--------------+------------+------------+-----------------+----------------+61|Configuration | Persistent | Uniform | st_ino == d_ino | d_ino == i_ino |62| | st_ino | st_dev | | [*] |63+==============+=====+======+=====+======+========+========+========+=======+64| | dir | !dir | dir | !dir | dir + !dir | dir | !dir |65+--------------+-----+------+-----+------+--------+--------+--------+-------+66| All layers | Y | Y | Y | Y | Y | Y | Y | Y |67| on same fs | | | | | | | | |68+--------------+-----+------+-----+------+--------+--------+--------+-------+69| Layers not | N | N | Y | N | N | Y | N | Y |70| on same fs, | | | | | | | | |71| xino=off | | | | | | | | |72+--------------+-----+------+-----+------+--------+--------+--------+-------+73| xino=on/auto | Y | Y | Y | Y | Y | Y | Y | Y |74+--------------+-----+------+-----+------+--------+--------+--------+-------+75| xino=on/auto,| N | N | Y | N | N | Y | N | Y |76| ino overflow | | | | | | | | |77+--------------+-----+------+-----+------+--------+--------+--------+-------+78 79[*] nfsd v3 readdirplus verifies d_ino == i_ino. i_ino is exposed via several80/proc files, such as /proc/locks and /proc/self/fdinfo/<fd> of an inotify81file descriptor.82 83Upper and Lower84---------------85 86An overlay filesystem combines two filesystems - an 'upper' filesystem87and a 'lower' filesystem. When a name exists in both filesystems, the88object in the 'upper' filesystem is visible while the object in the89'lower' filesystem is either hidden or, in the case of directories,90merged with the 'upper' object.91 92It would be more correct to refer to an upper and lower 'directory93tree' rather than 'filesystem' as it is quite possible for both94directory trees to be in the same filesystem and there is no95requirement that the root of a filesystem be given for either upper or96lower.97 98A wide range of filesystems supported by Linux can be the lower filesystem,99but not all filesystems that are mountable by Linux have the features100needed for OverlayFS to work. The lower filesystem does not need to be101writable. The lower filesystem can even be another overlayfs. The upper102filesystem will normally be writable and if it is it must support the103creation of trusted.* and/or user.* extended attributes, and must provide104valid d_type in readdir responses, so NFS is not suitable.105 106A read-only overlay of two read-only filesystems may use any107filesystem type.108 109Directories110-----------111 112Overlaying mainly involves directories. If a given name appears in both113upper and lower filesystems and refers to a non-directory in either,114then the lower object is hidden - the name refers only to the upper115object.116 117Where both upper and lower objects are directories, a merged directory118is formed.119 120At mount time, the two directories given as mount options "lowerdir" and121"upperdir" are combined into a merged directory::122 123 mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\124 workdir=/work /merged125 126The "workdir" needs to be an empty directory on the same filesystem127as upperdir.128 129Then whenever a lookup is requested in such a merged directory, the130lookup is performed in each actual directory and the combined result131is cached in the dentry belonging to the overlay filesystem. If both132actual lookups find directories, both are stored and a merged133directory is created, otherwise only one is stored: the upper if it134exists, else the lower.135 136Only the lists of names from directories are merged. Other content137such as metadata and extended attributes are reported for the upper138directory only. These attributes of the lower directory are hidden.139 140whiteouts and opaque directories141--------------------------------142 143In order to support rm and rmdir without changing the lower144filesystem, an overlay filesystem needs to record in the upper filesystem145that files have been removed. This is done using whiteouts and opaque146directories (non-directories are always opaque).147 148A whiteout is created as a character device with 0/0 device number or149as a zero-size regular file with the xattr "trusted.overlay.whiteout".150 151When a whiteout is found in the upper level of a merged directory, any152matching name in the lower level is ignored, and the whiteout itself153is also hidden.154 155A directory is made opaque by setting the xattr "trusted.overlay.opaque"156to "y". Where the upper filesystem contains an opaque directory, any157directory in the lower filesystem with the same name is ignored.158 159An opaque directory should not conntain any whiteouts, because they do not160serve any purpose. A merge directory containing regular files with the xattr161"trusted.overlay.whiteout", should be additionally marked by setting the xattr162"trusted.overlay.opaque" to "x" on the merge directory itself.163This is needed to avoid the overhead of checking the "trusted.overlay.whiteout"164on all entries during readdir in the common case.165 166readdir167-------168 169When a 'readdir' request is made on a merged directory, the upper and170lower directories are each read and the name lists merged in the171obvious way (upper is read first, then lower - entries that already172exist are not re-added). This merged name list is cached in the173'struct file' and so remains as long as the file is kept open. If the174directory is opened and read by two processes at the same time, they175will each have separate caches. A seekdir to the start of the176directory (offset 0) followed by a readdir will cause the cache to be177discarded and rebuilt.178 179This means that changes to the merged directory do not appear while a180directory is being read. This is unlikely to be noticed by many181programs.182 183seek offsets are assigned sequentially when the directories are read.184Thus if:185 186 - read part of a directory187 - remember an offset, and close the directory188 - re-open the directory some time later189 - seek to the remembered offset190 191there may be little correlation between the old and new locations in192the list of filenames, particularly if anything has changed in the193directory.194 195Readdir on directories that are not merged is simply handled by the196underlying directory (upper or lower).197 198renaming directories199--------------------200 201When renaming a directory that is on the lower layer or merged (i.e. the202directory was not created on the upper layer to start with) overlayfs can203handle it in two different ways:204 2051. return EXDEV error: this error is returned by rename(2) when trying to206 move a file or directory across filesystem boundaries. Hence207 applications are usually prepared to handle this error (mv(1) for example208 recursively copies the directory tree). This is the default behavior.209 2102. If the "redirect_dir" feature is enabled, then the directory will be211 copied up (but not the contents). Then the "trusted.overlay.redirect"212 extended attribute is set to the path of the original location from the213 root of the overlay. Finally the directory is moved to the new214 location.215 216There are several ways to tune the "redirect_dir" feature.217 218Kernel config options:219 220- OVERLAY_FS_REDIRECT_DIR:221 If this is enabled, then redirect_dir is turned on by default.222- OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW:223 If this is enabled, then redirects are always followed by default. Enabling224 this results in a less secure configuration. Enable this option only when225 worried about backward compatibility with kernels that have the redirect_dir226 feature and follow redirects even if turned off.227 228Module options (can also be changed through /sys/module/overlay/parameters/):229 230- "redirect_dir=BOOL":231 See OVERLAY_FS_REDIRECT_DIR kernel config option above.232- "redirect_always_follow=BOOL":233 See OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW kernel config option above.234- "redirect_max=NUM":235 The maximum number of bytes in an absolute redirect (default is 256).236 237Mount options:238 239- "redirect_dir=on":240 Redirects are enabled.241- "redirect_dir=follow":242 Redirects are not created, but followed.243- "redirect_dir=nofollow":244 Redirects are not created and not followed.245- "redirect_dir=off":246 If "redirect_always_follow" is enabled in the kernel/module config,247 this "off" translates to "follow", otherwise it translates to "nofollow".248 249When the NFS export feature is enabled, every copied up directory is250indexed by the file handle of the lower inode and a file handle of the251upper directory is stored in a "trusted.overlay.upper" extended attribute252on the index entry. On lookup of a merged directory, if the upper253directory does not match the file handle stores in the index, that is an254indication that multiple upper directories may be redirected to the same255lower directory. In that case, lookup returns an error and warns about256a possible inconsistency.257 258Because lower layer redirects cannot be verified with the index, enabling259NFS export support on an overlay filesystem with no upper layer requires260turning off redirect follow (e.g. "redirect_dir=nofollow").261 262 263Non-directories264---------------265 266Objects that are not directories (files, symlinks, device-special267files etc.) are presented either from the upper or lower filesystem as268appropriate. When a file in the lower filesystem is accessed in a way269the requires write-access, such as opening for write access, changing270some metadata etc., the file is first copied from the lower filesystem271to the upper filesystem (copy_up). Note that creating a hard-link272also requires copy_up, though of course creation of a symlink does273not.274 275The copy_up may turn out to be unnecessary, for example if the file is276opened for read-write but the data is not modified.277 278The copy_up process first makes sure that the containing directory279exists in the upper filesystem - creating it and any parents as280necessary. It then creates the object with the same metadata (owner,281mode, mtime, symlink-target etc.) and then if the object is a file, the282data is copied from the lower to the upper filesystem. Finally any283extended attributes are copied up.284 285Once the copy_up is complete, the overlay filesystem simply286provides direct access to the newly created file in the upper287filesystem - future operations on the file are barely noticed by the288overlay filesystem (though an operation on the name of the file such as289rename or unlink will of course be noticed and handled).290 291 292Permission model293----------------294 295Permission checking in the overlay filesystem follows these principles:296 297 1) permission check SHOULD return the same result before and after copy up298 299 2) task creating the overlay mount MUST NOT gain additional privileges300 301 3) non-mounting task MAY gain additional privileges through the overlay,302 compared to direct access on underlying lower or upper filesystems303 304This is achieved by performing two permission checks on each access:305 306 a) check if current task is allowed access based on local DAC (owner,307 group, mode and posix acl), as well as MAC checks308 309 b) check if mounting task would be allowed real operation on lower or310 upper layer based on underlying filesystem permissions, again including311 MAC checks312 313Check (a) ensures consistency (1) since owner, group, mode and posix acls314are copied up. On the other hand it can result in server enforced315permissions (used by NFS, for example) being ignored (3).316 317Check (b) ensures that no task gains permissions to underlying layers that318the mounting task does not have (2). This also means that it is possible319to create setups where the consistency rule (1) does not hold; normally,320however, the mounting task will have sufficient privileges to perform all321operations.322 323Another way to demonstrate this model is drawing parallels between::324 325 mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,... /merged326 327and::328 329 cp -a /lower /upper330 mount --bind /upper /merged331 332The resulting access permissions should be the same. The difference is in333the time of copy (on-demand vs. up-front).334 335 336Multiple lower layers337---------------------338 339Multiple lower layers can now be given using the colon (":") as a340separator character between the directory names. For example::341 342 mount -t overlay overlay -olowerdir=/lower1:/lower2:/lower3 /merged343 344As the example shows, "upperdir=" and "workdir=" may be omitted. In345that case the overlay will be read-only.346 347The specified lower directories will be stacked beginning from the348rightmost one and going left. In the above example lower1 will be the349top, lower2 the middle and lower3 the bottom layer.350 351Note: directory names containing colons can be provided as lower layer by352escaping the colons with a single backslash. For example::353 354 mount -t overlay overlay -olowerdir=/a\:lower\:\:dir /merged355 356Since kernel version v6.8, directory names containing colons can also357be configured as lower layer using the "lowerdir+" mount options and the358fsconfig syscall from new mount api. For example::359 360 fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerdir+", "/a:lower::dir", 0);361 362In the latter case, colons in lower layer directory names will be escaped363as an octal characters (\072) when displayed in /proc/self/mountinfo.364 365Metadata only copy up366---------------------367 368When the "metacopy" feature is enabled, overlayfs will only copy369up metadata (as opposed to whole file), when a metadata specific operation370like chown/chmod is performed. An upper file in this state is marked with371"trusted.overlayfs.metacopy" xattr which indicates that the upper file372contains no data. The data will be copied up later when file is opened for373WRITE operation. After the lower file's data is copied up,374the "trusted.overlayfs.metacopy" xattr is removed from the upper file.375 376In other words, this is delayed data copy up operation and data is copied377up when there is a need to actually modify data.378 379There are multiple ways to enable/disable this feature. A config option380CONFIG_OVERLAY_FS_METACOPY can be set/unset to enable/disable this feature381by default. Or one can enable/disable it at module load time with module382parameter metacopy=on/off. Lastly, there is also a per mount option383metacopy=on/off to enable/disable this feature per mount.384 385Do not use metacopy=on with untrusted upper/lower directories. Otherwise386it is possible that an attacker can create a handcrafted file with387appropriate REDIRECT and METACOPY xattrs, and gain access to file on lower388pointed by REDIRECT. This should not be possible on local system as setting389"trusted." xattrs will require CAP_SYS_ADMIN. But it should be possible390for untrusted layers like from a pen drive.391 392Note: redirect_dir={off|nofollow|follow[*]} and nfs_export=on mount options393conflict with metacopy=on, and will result in an error.394 395[*] redirect_dir=follow only conflicts with metacopy=on if upperdir=... is396given.397 398 399Data-only lower layers400----------------------401 402With "metacopy" feature enabled, an overlayfs regular file may be a composition403of information from up to three different layers:404 405 1) metadata from a file in the upper layer406 407 2) st_ino and st_dev object identifier from a file in a lower layer408 409 3) data from a file in another lower layer (further below)410 411The "lower data" file can be on any lower layer, except from the top most412lower layer.413 414Below the top most lower layer, any number of lower most layers may be defined415as "data-only" lower layers, using double colon ("::") separators.416A normal lower layer is not allowed to be below a data-only layer, so single417colon separators are not allowed to the right of double colon ("::") separators.418 419 420For example::421 422 mount -t overlay overlay -olowerdir=/l1:/l2:/l3::/do1::/do2 /merged423 424The paths of files in the "data-only" lower layers are not visible in the425merged overlayfs directories and the metadata and st_ino/st_dev of files426in the "data-only" lower layers are not visible in overlayfs inodes.427 428Only the data of the files in the "data-only" lower layers may be visible429when a "metacopy" file in one of the lower layers above it, has a "redirect"430to the absolute path of the "lower data" file in the "data-only" lower layer.431 432Since kernel version v6.8, "data-only" lower layers can also be added using433the "datadir+" mount options and the fsconfig syscall from new mount api.434For example::435 436 fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerdir+", "/l1", 0);437 fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerdir+", "/l2", 0);438 fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerdir+", "/l3", 0);439 fsconfig(fs_fd, FSCONFIG_SET_STRING, "datadir+", "/do1", 0);440 fsconfig(fs_fd, FSCONFIG_SET_STRING, "datadir+", "/do2", 0);441 442 443fs-verity support444-----------------445 446During metadata copy up of a lower file, if the source file has447fs-verity enabled and overlay verity support is enabled, then the448digest of the lower file is added to the "trusted.overlay.metacopy"449xattr. This is then used to verify the content of the lower file450each the time the metacopy file is opened.451 452When a layer containing verity xattrs is used, it means that any such453metacopy file in the upper layer is guaranteed to match the content454that was in the lower at the time of the copy-up. If at any time455(during a mount, after a remount, etc) such a file in the lower is456replaced or modified in any way, access to the corresponding file in457overlayfs will result in EIO errors (either on open, due to overlayfs458digest check, or from a later read due to fs-verity) and a detailed459error is printed to the kernel logs. For more details of how fs-verity460file access works, see :ref:`Documentation/filesystems/fsverity.rst461<accessing_verity_files>`.462 463Verity can be used as a general robustness check to detect accidental464changes in the overlayfs directories in use. But, with additional care465it can also give more powerful guarantees. For example, if the upper466layer is fully trusted (by using dm-verity or something similar), then467an untrusted lower layer can be used to supply validated file content468for all metacopy files. If additionally the untrusted lower469directories are specified as "Data-only", then they can only supply470such file content, and the entire mount can be trusted to match the471upper layer.472 473This feature is controlled by the "verity" mount option, which474supports these values:475 476- "off":477 The metacopy digest is never generated or used. This is the478 default if verity option is not specified.479- "on":480 Whenever a metacopy files specifies an expected digest, the481 corresponding data file must match the specified digest. When482 generating a metacopy file the verity digest will be set in it483 based on the source file (if it has one).484- "require":485 Same as "on", but additionally all metacopy files must specify a486 digest (or EIO is returned on open). This means metadata copy up487 will only be used if the data file has fs-verity enabled,488 otherwise a full copy-up is used.489 490Sharing and copying layers491--------------------------492 493Lower layers may be shared among several overlay mounts and that is indeed494a very common practice. An overlay mount may use the same lower layer495path as another overlay mount and it may use a lower layer path that is496beneath or above the path of another overlay lower layer path.497 498Using an upper layer path and/or a workdir path that are already used by499another overlay mount is not allowed and may fail with EBUSY. Using500partially overlapping paths is not allowed and may fail with EBUSY.501If files are accessed from two overlayfs mounts which share or overlap the502upper layer and/or workdir path the behavior of the overlay is undefined,503though it will not result in a crash or deadlock.504 505Mounting an overlay using an upper layer path, where the upper layer path506was previously used by another mounted overlay in combination with a507different lower layer path, is allowed, unless the "index" or "metacopy"508features are enabled.509 510With the "index" feature, on the first time mount, an NFS file511handle of the lower layer root directory, along with the UUID of the lower512filesystem, are encoded and stored in the "trusted.overlay.origin" extended513attribute on the upper layer root directory. On subsequent mount attempts,514the lower root directory file handle and lower filesystem UUID are compared515to the stored origin in upper root directory. On failure to verify the516lower root origin, mount will fail with ESTALE. An overlayfs mount with517"index" enabled will fail with EOPNOTSUPP if the lower filesystem518does not support NFS export, lower filesystem does not have a valid UUID or519if the upper filesystem does not support extended attributes.520 521For the "metacopy" feature, there is no verification mechanism at522mount time. So if same upper is mounted with different set of lower, mount523probably will succeed but expect the unexpected later on. So don't do it.524 525It is quite a common practice to copy overlay layers to a different526directory tree on the same or different underlying filesystem, and even527to a different machine. With the "index" feature, trying to mount528the copied layers will fail the verification of the lower root file handle.529 530Nesting overlayfs mounts531------------------------532 533It is possible to use a lower directory that is stored on an overlayfs534mount. For regular files this does not need any special care. However, files535that have overlayfs attributes, such as whiteouts or "overlay.*" xattrs will be536interpreted by the underlying overlayfs mount and stripped out. In order to537allow the second overlayfs mount to see the attributes they must be escaped.538 539Overlayfs specific xattrs are escaped by using a special prefix of540"overlay.overlay.". So, a file with a "trusted.overlay.overlay.metacopy" xattr541in the lower dir will be exposed as a regular file with a542"trusted.overlay.metacopy" xattr in the overlayfs mount. This can be nested by543repeating the prefix multiple time, as each instance only removes one prefix.544 545A lower dir with a regular whiteout will always be handled by the overlayfs546mount, so to support storing an effective whiteout file in an overlayfs mount an547alternative form of whiteout is supported. This form is a regular, zero-size548file with the "overlay.whiteout" xattr set, inside a directory with the549"overlay.opaque" xattr set to "x" (see `whiteouts and opaque directories`_).550These alternative whiteouts are never created by overlayfs, but can be used by551userspace tools (like containers) that generate lower layers.552These alternative whiteouts can be escaped using the standard xattr escape553mechanism in order to properly nest to any depth.554 555Non-standard behavior556---------------------557 558Current version of overlayfs can act as a mostly POSIX compliant559filesystem.560 561This is the list of cases that overlayfs doesn't currently handle:562 563 a) POSIX mandates updating st_atime for reads. This is currently not564 done in the case when the file resides on a lower layer.565 566 b) If a file residing on a lower layer is opened for read-only and then567 memory mapped with MAP_SHARED, then subsequent changes to the file are not568 reflected in the memory mapping.569 570 c) If a file residing on a lower layer is being executed, then opening that571 file for write or truncating the file will not be denied with ETXTBSY.572 573The following options allow overlayfs to act more like a standards574compliant filesystem:575 576redirect_dir577````````````578 579Enabled with the mount option or module option: "redirect_dir=on" or with580the kernel config option CONFIG_OVERLAY_FS_REDIRECT_DIR=y.581 582If this feature is disabled, then rename(2) on a lower or merged directory583will fail with EXDEV ("Invalid cross-device link").584 585index586`````587 588Enabled with the mount option or module option "index=on" or with the589kernel config option CONFIG_OVERLAY_FS_INDEX=y.590 591If this feature is disabled and a file with multiple hard links is copied592up, then this will "break" the link. Changes will not be propagated to593other names referring to the same inode.594 595xino596````597 598Enabled with the mount option "xino=auto" or "xino=on", with the module599option "xino_auto=on" or with the kernel config option600CONFIG_OVERLAY_FS_XINO_AUTO=y. Also implicitly enabled by using the same601underlying filesystem for all layers making up the overlay.602 603If this feature is disabled or the underlying filesystem doesn't have604enough free bits in the inode number, then overlayfs will not be able to605guarantee that the values of st_ino and st_dev returned by stat(2) and the606value of d_ino returned by readdir(3) will act like on a normal filesystem.607E.g. the value of st_dev may be different for two objects in the same608overlay filesystem and the value of st_ino for filesystem objects may not be609persistent and could change even while the overlay filesystem is mounted, as610summarized in the `Inode properties`_ table above.611 612 613Changes to underlying filesystems614---------------------------------615 616Changes to the underlying filesystems while part of a mounted overlay617filesystem are not allowed. If the underlying filesystem is changed,618the behavior of the overlay is undefined, though it will not result in619a crash or deadlock.620 621Offline changes, when the overlay is not mounted, are allowed to the622upper tree. Offline changes to the lower tree are only allowed if the623"metacopy", "index", "xino" and "redirect_dir" features624have not been used. If the lower tree is modified and any of these625features has been used, the behavior of the overlay is undefined,626though it will not result in a crash or deadlock.627 628When the overlay NFS export feature is enabled, overlay filesystems629behavior on offline changes of the underlying lower layer is different630than the behavior when NFS export is disabled.631 632On every copy_up, an NFS file handle of the lower inode, along with the633UUID of the lower filesystem, are encoded and stored in an extended634attribute "trusted.overlay.origin" on the upper inode.635 636When the NFS export feature is enabled, a lookup of a merged directory,637that found a lower directory at the lookup path or at the path pointed638to by the "trusted.overlay.redirect" extended attribute, will verify639that the found lower directory file handle and lower filesystem UUID640match the origin file handle that was stored at copy_up time. If a641found lower directory does not match the stored origin, that directory642will not be merged with the upper directory.643 644 645 646NFS export647----------648 649When the underlying filesystems supports NFS export and the "nfs_export"650feature is enabled, an overlay filesystem may be exported to NFS.651 652With the "nfs_export" feature, on copy_up of any lower object, an index653entry is created under the index directory. The index entry name is the654hexadecimal representation of the copy up origin file handle. For a655non-directory object, the index entry is a hard link to the upper inode.656For a directory object, the index entry has an extended attribute657"trusted.overlay.upper" with an encoded file handle of the upper658directory inode.659 660When encoding a file handle from an overlay filesystem object, the661following rules apply:662 663 1. For a non-upper object, encode a lower file handle from lower inode664 2. For an indexed object, encode a lower file handle from copy_up origin665 3. For a pure-upper object and for an existing non-indexed upper object,666 encode an upper file handle from upper inode667 668The encoded overlay file handle includes:669 670 - Header including path type information (e.g. lower/upper)671 - UUID of the underlying filesystem672 - Underlying filesystem encoding of underlying inode673 674This encoding format is identical to the encoding format file handles that675are stored in extended attribute "trusted.overlay.origin".676 677When decoding an overlay file handle, the following steps are followed:678 679 1. Find underlying layer by UUID and path type information.680 2. Decode the underlying filesystem file handle to underlying dentry.681 3. For a lower file handle, lookup the handle in index directory by name.682 4. If a whiteout is found in index, return ESTALE. This represents an683 overlay object that was deleted after its file handle was encoded.684 5. For a non-directory, instantiate a disconnected overlay dentry from the685 decoded underlying dentry, the path type and index inode, if found.686 6. For a directory, use the connected underlying decoded dentry, path type687 and index, to lookup a connected overlay dentry.688 689Decoding a non-directory file handle may return a disconnected dentry.690copy_up of that disconnected dentry will create an upper index entry with691no upper alias.692 693When overlay filesystem has multiple lower layers, a middle layer694directory may have a "redirect" to lower directory. Because middle layer695"redirects" are not indexed, a lower file handle that was encoded from the696"redirect" origin directory, cannot be used to find the middle or upper697layer directory. Similarly, a lower file handle that was encoded from a698descendant of the "redirect" origin directory, cannot be used to699reconstruct a connected overlay path. To mitigate the cases of700directories that cannot be decoded from a lower file handle, these701directories are copied up on encode and encoded as an upper file handle.702On an overlay filesystem with no upper layer this mitigation cannot be703used NFS export in this setup requires turning off redirect follow (e.g.704"redirect_dir=nofollow").705 706The overlay filesystem does not support non-directory connectable file707handles, so exporting with the 'subtree_check' exportfs configuration will708cause failures to lookup files over NFS.709 710When the NFS export feature is enabled, all directory index entries are711verified on mount time to check that upper file handles are not stale.712This verification may cause significant overhead in some cases.713 714Note: the mount options index=off,nfs_export=on are conflicting for a715read-write mount and will result in an error.716 717Note: the mount option uuid=off can be used to replace UUID of the underlying718filesystem in file handles with null, and effectively disable UUID checks. This719can be useful in case the underlying disk is copied and the UUID of this copy720is changed. This is only applicable if all lower/upper/work directories are on721the same filesystem, otherwise it will fallback to normal behaviour.722 723 724UUID and fsid725-------------726 727The UUID of overlayfs instance itself and the fsid reported by statfs(2) are728controlled by the "uuid" mount option, which supports these values:729 730- "null":731 UUID of overlayfs is null. fsid is taken from upper most filesystem.732- "off":733 UUID of overlayfs is null. fsid is taken from upper most filesystem.734 UUID of underlying layers is ignored.735- "on":736 UUID of overlayfs is generated and used to report a unique fsid.737 UUID is stored in xattr "trusted.overlay.uuid", making overlayfs fsid738 unique and persistent. This option requires an overlayfs with upper739 filesystem that supports xattrs.740- "auto": (default)741 UUID is taken from xattr "trusted.overlay.uuid" if it exists.742 Upgrade to "uuid=on" on first time mount of new overlay filesystem that743 meets the prerequites.744 Downgrade to "uuid=null" for existing overlay filesystems that were never745 mounted with "uuid=on".746 747 748Volatile mount749--------------750 751This is enabled with the "volatile" mount option. Volatile mounts are not752guaranteed to survive a crash. It is strongly recommended that volatile753mounts are only used if data written to the overlay can be recreated754without significant effort.755 756The advantage of mounting with the "volatile" option is that all forms of757sync calls to the upper filesystem are omitted.758 759In order to avoid a giving a false sense of safety, the syncfs (and fsync)760semantics of volatile mounts are slightly different than that of the rest of761VFS. If any writeback error occurs on the upperdir's filesystem after a762volatile mount takes place, all sync functions will return an error. Once this763condition is reached, the filesystem will not recover, and every subsequent sync764call will return an error, even if the upperdir has not experience a new error765since the last sync call.766 767When overlay is mounted with "volatile" option, the directory768"$workdir/work/incompat/volatile" is created. During next mount, overlay769checks for this directory and refuses to mount if present. This is a strong770indicator that user should throw away upper and work directories and create771fresh one. In very limited cases where the user knows that the system has772not crashed and contents of upperdir are intact, The "volatile" directory773can be removed.774 775 776User xattr777----------778 779The "-o userxattr" mount option forces overlayfs to use the780"user.overlay." xattr namespace instead of "trusted.overlay.". This is781useful for unprivileged mounting of overlayfs.782 783 784Testsuite785---------786 787There's a testsuite originally developed by David Howells and currently788maintained by Amir Goldstein at:789 790https://github.com/amir73il/unionmount-testsuite.git791 792Run as root::793 794 # cd unionmount-testsuite795 # ./run --ov --verify796