908 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3.. _fsverity:4 5=======================================================6fs-verity: read-only file-based authenticity protection7=======================================================8 9Introduction10============11 12fs-verity (``fs/verity/``) is a support layer that filesystems can13hook into to support transparent integrity and authenticity protection14of read-only files. Currently, it is supported by the ext4, f2fs, and15btrfs filesystems. Like fscrypt, not too much filesystem-specific16code is needed to support fs-verity.17 18fs-verity is similar to `dm-verity19<https://www.kernel.org/doc/Documentation/device-mapper/verity.txt>`_20but works on files rather than block devices. On regular files on21filesystems supporting fs-verity, userspace can execute an ioctl that22causes the filesystem to build a Merkle tree for the file and persist23it to a filesystem-specific location associated with the file.24 25After this, the file is made readonly, and all reads from the file are26automatically verified against the file's Merkle tree. Reads of any27corrupted data, including mmap reads, will fail.28 29Userspace can use another ioctl to retrieve the root hash (actually30the "fs-verity file digest", which is a hash that includes the Merkle31tree root hash) that fs-verity is enforcing for the file. This ioctl32executes in constant time, regardless of the file size.33 34fs-verity is essentially a way to hash a file in constant time,35subject to the caveat that reads which would violate the hash will36fail at runtime.37 38Use cases39=========40 41By itself, fs-verity only provides integrity protection, i.e.42detection of accidental (non-malicious) corruption.43 44However, because fs-verity makes retrieving the file hash extremely45efficient, it's primarily meant to be used as a tool to support46authentication (detection of malicious modifications) or auditing47(logging file hashes before use).48 49A standard file hash could be used instead of fs-verity. However,50this is inefficient if the file is large and only a small portion may51be accessed. This is often the case for Android application package52(APK) files, for example. These typically contain many translations,53classes, and other resources that are infrequently or even never54accessed on a particular device. It would be slow and wasteful to55read and hash the entire file before starting the application.56 57Unlike an ahead-of-time hash, fs-verity also re-verifies data each58time it's paged in. This ensures that malicious disk firmware can't59undetectably change the contents of the file at runtime.60 61fs-verity does not replace or obsolete dm-verity. dm-verity should62still be used on read-only filesystems. fs-verity is for files that63must live on a read-write filesystem because they are independently64updated and potentially user-installed, so dm-verity cannot be used.65 66fs-verity does not mandate a particular scheme for authenticating its67file hashes. (Similarly, dm-verity does not mandate a particular68scheme for authenticating its block device root hashes.) Options for69authenticating fs-verity file hashes include:70 71- Trusted userspace code. Often, the userspace code that accesses72 files can be trusted to authenticate them. Consider e.g. an73 application that wants to authenticate data files before using them,74 or an application loader that is part of the operating system (which75 is already authenticated in a different way, such as by being loaded76 from a read-only partition that uses dm-verity) and that wants to77 authenticate applications before loading them. In these cases, this78 trusted userspace code can authenticate a file's contents by79 retrieving its fs-verity digest using `FS_IOC_MEASURE_VERITY`_, then80 verifying a signature of it using any userspace cryptographic81 library that supports digital signatures.82 83- Integrity Measurement Architecture (IMA). IMA supports fs-verity84 file digests as an alternative to its traditional full file digests.85 "IMA appraisal" enforces that files contain a valid, matching86 signature in their "security.ima" extended attribute, as controlled87 by the IMA policy. For more information, see the IMA documentation.88 89- Integrity Policy Enforcement (IPE). IPE supports enforcing access90 control decisions based on immutable security properties of files,91 including those protected by fs-verity's built-in signatures.92 "IPE policy" specifically allows for the authorization of fs-verity93 files using properties ``fsverity_digest`` for identifying94 files by their verity digest, and ``fsverity_signature`` to authorize95 files with a verified fs-verity's built-in signature. For96 details on configuring IPE policies and understanding its operational97 modes, please refer to :doc:`IPE admin guide </admin-guide/LSM/ipe>`.98 99- Trusted userspace code in combination with `Built-in signature100 verification`_. This approach should be used only with great care.101 102User API103========104 105FS_IOC_ENABLE_VERITY106--------------------107 108The FS_IOC_ENABLE_VERITY ioctl enables fs-verity on a file. It takes109in a pointer to a struct fsverity_enable_arg, defined as110follows::111 112 struct fsverity_enable_arg {113 __u32 version;114 __u32 hash_algorithm;115 __u32 block_size;116 __u32 salt_size;117 __u64 salt_ptr;118 __u32 sig_size;119 __u32 __reserved1;120 __u64 sig_ptr;121 __u64 __reserved2[11];122 };123 124This structure contains the parameters of the Merkle tree to build for125the file. It must be initialized as follows:126 127- ``version`` must be 1.128- ``hash_algorithm`` must be the identifier for the hash algorithm to129 use for the Merkle tree, such as FS_VERITY_HASH_ALG_SHA256. See130 ``include/uapi/linux/fsverity.h`` for the list of possible values.131- ``block_size`` is the Merkle tree block size, in bytes. In Linux132 v6.3 and later, this can be any power of 2 between (inclusively)133 1024 and the minimum of the system page size and the filesystem134 block size. In earlier versions, the page size was the only allowed135 value.136- ``salt_size`` is the size of the salt in bytes, or 0 if no salt is137 provided. The salt is a value that is prepended to every hashed138 block; it can be used to personalize the hashing for a particular139 file or device. Currently the maximum salt size is 32 bytes.140- ``salt_ptr`` is the pointer to the salt, or NULL if no salt is141 provided.142- ``sig_size`` is the size of the builtin signature in bytes, or 0 if no143 builtin signature is provided. Currently the builtin signature is144 (somewhat arbitrarily) limited to 16128 bytes.145- ``sig_ptr`` is the pointer to the builtin signature, or NULL if no146 builtin signature is provided. A builtin signature is only needed147 if the `Built-in signature verification`_ feature is being used. It148 is not needed for IMA appraisal, and it is not needed if the file149 signature is being handled entirely in userspace.150- All reserved fields must be zeroed.151 152FS_IOC_ENABLE_VERITY causes the filesystem to build a Merkle tree for153the file and persist it to a filesystem-specific location associated154with the file, then mark the file as a verity file. This ioctl may155take a long time to execute on large files, and it is interruptible by156fatal signals.157 158FS_IOC_ENABLE_VERITY checks for write access to the inode. However,159it must be executed on an O_RDONLY file descriptor and no processes160can have the file open for writing. Attempts to open the file for161writing while this ioctl is executing will fail with ETXTBSY. (This162is necessary to guarantee that no writable file descriptors will exist163after verity is enabled, and to guarantee that the file's contents are164stable while the Merkle tree is being built over it.)165 166On success, FS_IOC_ENABLE_VERITY returns 0, and the file becomes a167verity file. On failure (including the case of interruption by a168fatal signal), no changes are made to the file.169 170FS_IOC_ENABLE_VERITY can fail with the following errors:171 172- ``EACCES``: the process does not have write access to the file173- ``EBADMSG``: the builtin signature is malformed174- ``EBUSY``: this ioctl is already running on the file175- ``EEXIST``: the file already has verity enabled176- ``EFAULT``: the caller provided inaccessible memory177- ``EFBIG``: the file is too large to enable verity on178- ``EINTR``: the operation was interrupted by a fatal signal179- ``EINVAL``: unsupported version, hash algorithm, or block size; or180 reserved bits are set; or the file descriptor refers to neither a181 regular file nor a directory.182- ``EISDIR``: the file descriptor refers to a directory183- ``EKEYREJECTED``: the builtin signature doesn't match the file184- ``EMSGSIZE``: the salt or builtin signature is too long185- ``ENOKEY``: the ".fs-verity" keyring doesn't contain the certificate186 needed to verify the builtin signature187- ``ENOPKG``: fs-verity recognizes the hash algorithm, but it's not188 available in the kernel's crypto API as currently configured (e.g.189 for SHA-512, missing CONFIG_CRYPTO_SHA512).190- ``ENOTTY``: this type of filesystem does not implement fs-verity191- ``EOPNOTSUPP``: the kernel was not configured with fs-verity192 support; or the filesystem superblock has not had the 'verity'193 feature enabled on it; or the filesystem does not support fs-verity194 on this file. (See `Filesystem support`_.)195- ``EPERM``: the file is append-only; or, a builtin signature is196 required and one was not provided.197- ``EROFS``: the filesystem is read-only198- ``ETXTBSY``: someone has the file open for writing. This can be the199 caller's file descriptor, another open file descriptor, or the file200 reference held by a writable memory map.201 202FS_IOC_MEASURE_VERITY203---------------------204 205The FS_IOC_MEASURE_VERITY ioctl retrieves the digest of a verity file.206The fs-verity file digest is a cryptographic digest that identifies207the file contents that are being enforced on reads; it is computed via208a Merkle tree and is different from a traditional full-file digest.209 210This ioctl takes in a pointer to a variable-length structure::211 212 struct fsverity_digest {213 __u16 digest_algorithm;214 __u16 digest_size; /* input/output */215 __u8 digest[];216 };217 218``digest_size`` is an input/output field. On input, it must be219initialized to the number of bytes allocated for the variable-length220``digest`` field.221 222On success, 0 is returned and the kernel fills in the structure as223follows:224 225- ``digest_algorithm`` will be the hash algorithm used for the file226 digest. It will match ``fsverity_enable_arg::hash_algorithm``.227- ``digest_size`` will be the size of the digest in bytes, e.g. 32228 for SHA-256. (This can be redundant with ``digest_algorithm``.)229- ``digest`` will be the actual bytes of the digest.230 231FS_IOC_MEASURE_VERITY is guaranteed to execute in constant time,232regardless of the size of the file.233 234FS_IOC_MEASURE_VERITY can fail with the following errors:235 236- ``EFAULT``: the caller provided inaccessible memory237- ``ENODATA``: the file is not a verity file238- ``ENOTTY``: this type of filesystem does not implement fs-verity239- ``EOPNOTSUPP``: the kernel was not configured with fs-verity240 support, or the filesystem superblock has not had the 'verity'241 feature enabled on it. (See `Filesystem support`_.)242- ``EOVERFLOW``: the digest is longer than the specified243 ``digest_size`` bytes. Try providing a larger buffer.244 245FS_IOC_READ_VERITY_METADATA246---------------------------247 248The FS_IOC_READ_VERITY_METADATA ioctl reads verity metadata from a249verity file. This ioctl is available since Linux v5.12.250 251This ioctl allows writing a server program that takes a verity file252and serves it to a client program, such that the client can do its own253fs-verity compatible verification of the file. This only makes sense254if the client doesn't trust the server and if the server needs to255provide the storage for the client.256 257This is a fairly specialized use case, and most fs-verity users won't258need this ioctl.259 260This ioctl takes in a pointer to the following structure::261 262 #define FS_VERITY_METADATA_TYPE_MERKLE_TREE 1263 #define FS_VERITY_METADATA_TYPE_DESCRIPTOR 2264 #define FS_VERITY_METADATA_TYPE_SIGNATURE 3265 266 struct fsverity_read_metadata_arg {267 __u64 metadata_type;268 __u64 offset;269 __u64 length;270 __u64 buf_ptr;271 __u64 __reserved;272 };273 274``metadata_type`` specifies the type of metadata to read:275 276- ``FS_VERITY_METADATA_TYPE_MERKLE_TREE`` reads the blocks of the277 Merkle tree. The blocks are returned in order from the root level278 to the leaf level. Within each level, the blocks are returned in279 the same order that their hashes are themselves hashed.280 See `Merkle tree`_ for more information.281 282- ``FS_VERITY_METADATA_TYPE_DESCRIPTOR`` reads the fs-verity283 descriptor. See `fs-verity descriptor`_.284 285- ``FS_VERITY_METADATA_TYPE_SIGNATURE`` reads the builtin signature286 which was passed to FS_IOC_ENABLE_VERITY, if any. See `Built-in287 signature verification`_.288 289The semantics are similar to those of ``pread()``. ``offset``290specifies the offset in bytes into the metadata item to read from, and291``length`` specifies the maximum number of bytes to read from the292metadata item. ``buf_ptr`` is the pointer to the buffer to read into,293cast to a 64-bit integer. ``__reserved`` must be 0. On success, the294number of bytes read is returned. 0 is returned at the end of the295metadata item. The returned length may be less than ``length``, for296example if the ioctl is interrupted.297 298The metadata returned by FS_IOC_READ_VERITY_METADATA isn't guaranteed299to be authenticated against the file digest that would be returned by300`FS_IOC_MEASURE_VERITY`_, as the metadata is expected to be used to301implement fs-verity compatible verification anyway (though absent a302malicious disk, the metadata will indeed match). E.g. to implement303this ioctl, the filesystem is allowed to just read the Merkle tree304blocks from disk without actually verifying the path to the root node.305 306FS_IOC_READ_VERITY_METADATA can fail with the following errors:307 308- ``EFAULT``: the caller provided inaccessible memory309- ``EINTR``: the ioctl was interrupted before any data was read310- ``EINVAL``: reserved fields were set, or ``offset + length``311 overflowed312- ``ENODATA``: the file is not a verity file, or313 FS_VERITY_METADATA_TYPE_SIGNATURE was requested but the file doesn't314 have a builtin signature315- ``ENOTTY``: this type of filesystem does not implement fs-verity, or316 this ioctl is not yet implemented on it317- ``EOPNOTSUPP``: the kernel was not configured with fs-verity318 support, or the filesystem superblock has not had the 'verity'319 feature enabled on it. (See `Filesystem support`_.)320 321FS_IOC_GETFLAGS322---------------323 324The existing ioctl FS_IOC_GETFLAGS (which isn't specific to fs-verity)325can also be used to check whether a file has fs-verity enabled or not.326To do so, check for FS_VERITY_FL (0x00100000) in the returned flags.327 328The verity flag is not settable via FS_IOC_SETFLAGS. You must use329FS_IOC_ENABLE_VERITY instead, since parameters must be provided.330 331statx332-----333 334Since Linux v5.5, the statx() system call sets STATX_ATTR_VERITY if335the file has fs-verity enabled. This can perform better than336FS_IOC_GETFLAGS and FS_IOC_MEASURE_VERITY because it doesn't require337opening the file, and opening verity files can be expensive.338 339.. _accessing_verity_files:340 341Accessing verity files342======================343 344Applications can transparently access a verity file just like a345non-verity one, with the following exceptions:346 347- Verity files are readonly. They cannot be opened for writing or348 truncate()d, even if the file mode bits allow it. Attempts to do349 one of these things will fail with EPERM. However, changes to350 metadata such as owner, mode, timestamps, and xattrs are still351 allowed, since these are not measured by fs-verity. Verity files352 can also still be renamed, deleted, and linked to.353 354- Direct I/O is not supported on verity files. Attempts to use direct355 I/O on such files will fall back to buffered I/O.356 357- DAX (Direct Access) is not supported on verity files, because this358 would circumvent the data verification.359 360- Reads of data that doesn't match the verity Merkle tree will fail361 with EIO (for read()) or SIGBUS (for mmap() reads).362 363- If the sysctl "fs.verity.require_signatures" is set to 1 and the364 file is not signed by a key in the ".fs-verity" keyring, then365 opening the file will fail. See `Built-in signature verification`_.366 367Direct access to the Merkle tree is not supported. Therefore, if a368verity file is copied, or is backed up and restored, then it will lose369its "verity"-ness. fs-verity is primarily meant for files like370executables that are managed by a package manager.371 372File digest computation373=======================374 375This section describes how fs-verity hashes the file contents using a376Merkle tree to produce the digest which cryptographically identifies377the file contents. This algorithm is the same for all filesystems378that support fs-verity.379 380Userspace only needs to be aware of this algorithm if it needs to381compute fs-verity file digests itself, e.g. in order to sign files.382 383.. _fsverity_merkle_tree:384 385Merkle tree386-----------387 388The file contents is divided into blocks, where the block size is389configurable but is usually 4096 bytes. The end of the last block is390zero-padded if needed. Each block is then hashed, producing the first391level of hashes. Then, the hashes in this first level are grouped392into 'blocksize'-byte blocks (zero-padding the ends as needed) and393these blocks are hashed, producing the second level of hashes. This394proceeds up the tree until only a single block remains. The hash of395this block is the "Merkle tree root hash".396 397If the file fits in one block and is nonempty, then the "Merkle tree398root hash" is simply the hash of the single data block. If the file399is empty, then the "Merkle tree root hash" is all zeroes.400 401The "blocks" here are not necessarily the same as "filesystem blocks".402 403If a salt was specified, then it's zero-padded to the closest multiple404of the input size of the hash algorithm's compression function, e.g.40564 bytes for SHA-256 or 128 bytes for SHA-512. The padded salt is406prepended to every data or Merkle tree block that is hashed.407 408The purpose of the block padding is to cause every hash to be taken409over the same amount of data, which simplifies the implementation and410keeps open more possibilities for hardware acceleration. The purpose411of the salt padding is to make the salting "free" when the salted hash412state is precomputed, then imported for each hash.413 414Example: in the recommended configuration of SHA-256 and 4K blocks,415128 hash values fit in each block. Thus, each level of the Merkle416tree is approximately 128 times smaller than the previous, and for417large files the Merkle tree's size converges to approximately 1/127 of418the original file size. However, for small files, the padding is419significant, making the space overhead proportionally more.420 421.. _fsverity_descriptor:422 423fs-verity descriptor424--------------------425 426By itself, the Merkle tree root hash is ambiguous. For example, it427can't a distinguish a large file from a small second file whose data428is exactly the top-level hash block of the first file. Ambiguities429also arise from the convention of padding to the next block boundary.430 431To solve this problem, the fs-verity file digest is actually computed432as a hash of the following structure, which contains the Merkle tree433root hash as well as other fields such as the file size::434 435 struct fsverity_descriptor {436 __u8 version; /* must be 1 */437 __u8 hash_algorithm; /* Merkle tree hash algorithm */438 __u8 log_blocksize; /* log2 of size of data and tree blocks */439 __u8 salt_size; /* size of salt in bytes; 0 if none */440 __le32 __reserved_0x04; /* must be 0 */441 __le64 data_size; /* size of file the Merkle tree is built over */442 __u8 root_hash[64]; /* Merkle tree root hash */443 __u8 salt[32]; /* salt prepended to each hashed block */444 __u8 __reserved[144]; /* must be 0's */445 };446 447Built-in signature verification448===============================449 450CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y adds supports for in-kernel451verification of fs-verity builtin signatures.452 453**IMPORTANT**! Please take great care before using this feature.454It is not the only way to do signatures with fs-verity, and the455alternatives (such as userspace signature verification, and IMA456appraisal) can be much better. It's also easy to fall into a trap457of thinking this feature solves more problems than it actually does.458 459Enabling this option adds the following:460 4611. At boot time, the kernel creates a keyring named ".fs-verity". The462 root user can add trusted X.509 certificates to this keyring using463 the add_key() system call.464 4652. `FS_IOC_ENABLE_VERITY`_ accepts a pointer to a PKCS#7 formatted466 detached signature in DER format of the file's fs-verity digest.467 On success, the ioctl persists the signature alongside the Merkle468 tree. Then, any time the file is opened, the kernel verifies the469 file's actual digest against this signature, using the certificates470 in the ".fs-verity" keyring. This verification happens as long as the471 file's signature exists, regardless of the state of the sysctl variable472 "fs.verity.require_signatures" described in the next item. The IPE LSM473 relies on this behavior to recognize and label fsverity files474 that contain a verified built-in fsverity signature.475 4763. A new sysctl "fs.verity.require_signatures" is made available.477 When set to 1, the kernel requires that all verity files have a478 correctly signed digest as described in (2).479 480The data that the signature as described in (2) must be a signature of481is the fs-verity file digest in the following format::482 483 struct fsverity_formatted_digest {484 char magic[8]; /* must be "FSVerity" */485 __le16 digest_algorithm;486 __le16 digest_size;487 __u8 digest[];488 };489 490That's it. It should be emphasized again that fs-verity builtin491signatures are not the only way to do signatures with fs-verity. See492`Use cases`_ for an overview of ways in which fs-verity can be used.493fs-verity builtin signatures have some major limitations that should494be carefully considered before using them:495 496- Builtin signature verification does *not* make the kernel enforce497 that any files actually have fs-verity enabled. Thus, it is not a498 complete authentication policy. Currently, if it is used, one499 way to complete the authentication policy is for trusted userspace500 code to explicitly check whether files have fs-verity enabled with a501 signature before they are accessed. (With502 fs.verity.require_signatures=1, just checking whether fs-verity is503 enabled suffices.) But, in this case the trusted userspace code504 could just store the signature alongside the file and verify it505 itself using a cryptographic library, instead of using this feature.506 507- Another approach is to utilize fs-verity builtin signature508 verification in conjunction with the IPE LSM, which supports defining509 a kernel-enforced, system-wide authentication policy that allows only510 files with a verified fs-verity builtin signature to perform certain511 operations, such as execution. Note that IPE doesn't require512 fs.verity.require_signatures=1.513 Please refer to :doc:`IPE admin guide </admin-guide/LSM/ipe>` for514 more details.515 516- A file's builtin signature can only be set at the same time that517 fs-verity is being enabled on the file. Changing or deleting the518 builtin signature later requires re-creating the file.519 520- Builtin signature verification uses the same set of public keys for521 all fs-verity enabled files on the system. Different keys cannot be522 trusted for different files; each key is all or nothing.523 524- The sysctl fs.verity.require_signatures applies system-wide.525 Setting it to 1 only works when all users of fs-verity on the system526 agree that it should be set to 1. This limitation can prevent527 fs-verity from being used in cases where it would be helpful.528 529- Builtin signature verification can only use signature algorithms530 that are supported by the kernel. For example, the kernel does not531 yet support Ed25519, even though this is often the signature532 algorithm that is recommended for new cryptographic designs.533 534- fs-verity builtin signatures are in PKCS#7 format, and the public535 keys are in X.509 format. These formats are commonly used,536 including by some other kernel features (which is why the fs-verity537 builtin signatures use them), and are very feature rich.538 Unfortunately, history has shown that code that parses and handles539 these formats (which are from the 1990s and are based on ASN.1)540 often has vulnerabilities as a result of their complexity. This541 complexity is not inherent to the cryptography itself.542 543 fs-verity users who do not need advanced features of X.509 and544 PKCS#7 should strongly consider using simpler formats, such as plain545 Ed25519 keys and signatures, and verifying signatures in userspace.546 547 fs-verity users who choose to use X.509 and PKCS#7 anyway should548 still consider that verifying those signatures in userspace is more549 flexible (for other reasons mentioned earlier in this document) and550 eliminates the need to enable CONFIG_FS_VERITY_BUILTIN_SIGNATURES551 and its associated increase in kernel attack surface. In some cases552 it can even be necessary, since advanced X.509 and PKCS#7 features553 do not always work as intended with the kernel. For example, the554 kernel does not check X.509 certificate validity times.555 556 Note: IMA appraisal, which supports fs-verity, does not use PKCS#7557 for its signatures, so it partially avoids the issues discussed558 here. IMA appraisal does use X.509.559 560Filesystem support561==================562 563fs-verity is supported by several filesystems, described below. The564CONFIG_FS_VERITY kconfig option must be enabled to use fs-verity on565any of these filesystems.566 567``include/linux/fsverity.h`` declares the interface between the568``fs/verity/`` support layer and filesystems. Briefly, filesystems569must provide an ``fsverity_operations`` structure that provides570methods to read and write the verity metadata to a filesystem-specific571location, including the Merkle tree blocks and572``fsverity_descriptor``. Filesystems must also call functions in573``fs/verity/`` at certain times, such as when a file is opened or when574pages have been read into the pagecache. (See `Verifying data`_.)575 576ext4577----578 579ext4 supports fs-verity since Linux v5.4 and e2fsprogs v1.45.2.580 581To create verity files on an ext4 filesystem, the filesystem must have582been formatted with ``-O verity`` or had ``tune2fs -O verity`` run on583it. "verity" is an RO_COMPAT filesystem feature, so once set, old584kernels will only be able to mount the filesystem readonly, and old585versions of e2fsck will be unable to check the filesystem.586 587Originally, an ext4 filesystem with the "verity" feature could only be588mounted when its block size was equal to the system page size589(typically 4096 bytes). In Linux v6.3, this limitation was removed.590 591ext4 sets the EXT4_VERITY_FL on-disk inode flag on verity files. It592can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be cleared.593 594ext4 also supports encryption, which can be used simultaneously with595fs-verity. In this case, the plaintext data is verified rather than596the ciphertext. This is necessary in order to make the fs-verity file597digest meaningful, since every file is encrypted differently.598 599ext4 stores the verity metadata (Merkle tree and fsverity_descriptor)600past the end of the file, starting at the first 64K boundary beyond601i_size. This approach works because (a) verity files are readonly,602and (b) pages fully beyond i_size aren't visible to userspace but can603be read/written internally by ext4 with only some relatively small604changes to ext4. This approach avoids having to depend on the605EA_INODE feature and on rearchitecturing ext4's xattr support to606support paging multi-gigabyte xattrs into memory, and to support607encrypting xattrs. Note that the verity metadata *must* be encrypted608when the file is, since it contains hashes of the plaintext data.609 610ext4 only allows verity on extent-based files.611 612f2fs613----614 615f2fs supports fs-verity since Linux v5.4 and f2fs-tools v1.11.0.616 617To create verity files on an f2fs filesystem, the filesystem must have618been formatted with ``-O verity``.619 620f2fs sets the FADVISE_VERITY_BIT on-disk inode flag on verity files.621It can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be622cleared.623 624Like ext4, f2fs stores the verity metadata (Merkle tree and625fsverity_descriptor) past the end of the file, starting at the first62664K boundary beyond i_size. See explanation for ext4 above.627Moreover, f2fs supports at most 4096 bytes of xattr entries per inode628which usually wouldn't be enough for even a single Merkle tree block.629 630f2fs doesn't support enabling verity on files that currently have631atomic or volatile writes pending.632 633btrfs634-----635 636btrfs supports fs-verity since Linux v5.15. Verity-enabled inodes are637marked with a RO_COMPAT inode flag, and the verity metadata is stored638in separate btree items.639 640Implementation details641======================642 643Verifying data644--------------645 646fs-verity ensures that all reads of a verity file's data are verified,647regardless of which syscall is used to do the read (e.g. mmap(),648read(), pread()) and regardless of whether it's the first read or a649later read (unless the later read can return cached data that was650already verified). Below, we describe how filesystems implement this.651 652Pagecache653~~~~~~~~~654 655For filesystems using Linux's pagecache, the ``->read_folio()`` and656``->readahead()`` methods must be modified to verify folios before657they are marked Uptodate. Merely hooking ``->read_iter()`` would be658insufficient, since ``->read_iter()`` is not used for memory maps.659 660Therefore, fs/verity/ provides the function fsverity_verify_blocks()661which verifies data that has been read into the pagecache of a verity662inode. The containing folio must still be locked and not Uptodate, so663it's not yet readable by userspace. As needed to do the verification,664fsverity_verify_blocks() will call back into the filesystem to read665hash blocks via fsverity_operations::read_merkle_tree_page().666 667fsverity_verify_blocks() returns false if verification failed; in this668case, the filesystem must not set the folio Uptodate. Following this,669as per the usual Linux pagecache behavior, attempts by userspace to670read() from the part of the file containing the folio will fail with671EIO, and accesses to the folio within a memory map will raise SIGBUS.672 673In principle, verifying a data block requires verifying the entire674path in the Merkle tree from the data block to the root hash.675However, for efficiency the filesystem may cache the hash blocks.676Therefore, fsverity_verify_blocks() only ascends the tree reading hash677blocks until an already-verified hash block is seen. It then verifies678the path to that block.679 680This optimization, which is also used by dm-verity, results in681excellent sequential read performance. This is because usually (e.g.682127 in 128 times for 4K blocks and SHA-256) the hash block from the683bottom level of the tree will already be cached and checked from684reading a previous data block. However, random reads perform worse.685 686Block device based filesystems687~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~688 689Block device based filesystems (e.g. ext4 and f2fs) in Linux also use690the pagecache, so the above subsection applies too. However, they691also usually read many data blocks from a file at once, grouped into a692structure called a "bio". To make it easier for these types of693filesystems to support fs-verity, fs/verity/ also provides a function694fsverity_verify_bio() which verifies all data blocks in a bio.695 696ext4 and f2fs also support encryption. If a verity file is also697encrypted, the data must be decrypted before being verified. To698support this, these filesystems allocate a "post-read context" for699each bio and store it in ``->bi_private``::700 701 struct bio_post_read_ctx {702 struct bio *bio;703 struct work_struct work;704 unsigned int cur_step;705 unsigned int enabled_steps;706 };707 708``enabled_steps`` is a bitmask that specifies whether decryption,709verity, or both is enabled. After the bio completes, for each needed710postprocessing step the filesystem enqueues the bio_post_read_ctx on a711workqueue, and then the workqueue work does the decryption or712verification. Finally, folios where no decryption or verity error713occurred are marked Uptodate, and the folios are unlocked.714 715On many filesystems, files can contain holes. Normally,716``->readahead()`` simply zeroes hole blocks and considers the717corresponding data to be up-to-date; no bios are issued. To prevent718this case from bypassing fs-verity, filesystems use719fsverity_verify_blocks() to verify hole blocks.720 721Filesystems also disable direct I/O on verity files, since otherwise722direct I/O would bypass fs-verity.723 724Userspace utility725=================726 727This document focuses on the kernel, but a userspace utility for728fs-verity can be found at:729 730 https://git.kernel.org/pub/scm/fs/fsverity/fsverity-utils.git731 732See the README.md file in the fsverity-utils source tree for details,733including examples of setting up fs-verity protected files.734 735Tests736=====737 738To test fs-verity, use xfstests. For example, using `kvm-xfstests739<https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md>`_::740 741 kvm-xfstests -c ext4,f2fs,btrfs -g verity742 743FAQ744===745 746This section answers frequently asked questions about fs-verity that747weren't already directly answered in other parts of this document.748 749:Q: Why isn't fs-verity part of IMA?750:A: fs-verity and IMA (Integrity Measurement Architecture) have751 different focuses. fs-verity is a filesystem-level mechanism for752 hashing individual files using a Merkle tree. In contrast, IMA753 specifies a system-wide policy that specifies which files are754 hashed and what to do with those hashes, such as log them,755 authenticate them, or add them to a measurement list.756 757 IMA supports the fs-verity hashing mechanism as an alternative758 to full file hashes, for those who want the performance and759 security benefits of the Merkle tree based hash. However, it760 doesn't make sense to force all uses of fs-verity to be through761 IMA. fs-verity already meets many users' needs even as a762 standalone filesystem feature, and it's testable like other763 filesystem features e.g. with xfstests.764 765:Q: Isn't fs-verity useless because the attacker can just modify the766 hashes in the Merkle tree, which is stored on-disk?767:A: To verify the authenticity of an fs-verity file you must verify768 the authenticity of the "fs-verity file digest", which769 incorporates the root hash of the Merkle tree. See `Use cases`_.770 771:Q: Isn't fs-verity useless because the attacker can just replace a772 verity file with a non-verity one?773:A: See `Use cases`_. In the initial use case, it's really trusted774 userspace code that authenticates the files; fs-verity is just a775 tool to do this job efficiently and securely. The trusted776 userspace code will consider non-verity files to be inauthentic.777 778:Q: Why does the Merkle tree need to be stored on-disk? Couldn't you779 store just the root hash?780:A: If the Merkle tree wasn't stored on-disk, then you'd have to781 compute the entire tree when the file is first accessed, even if782 just one byte is being read. This is a fundamental consequence of783 how Merkle tree hashing works. To verify a leaf node, you need to784 verify the whole path to the root hash, including the root node785 (the thing which the root hash is a hash of). But if the root786 node isn't stored on-disk, you have to compute it by hashing its787 children, and so on until you've actually hashed the entire file.788 789 That defeats most of the point of doing a Merkle tree-based hash,790 since if you have to hash the whole file ahead of time anyway,791 then you could simply do sha256(file) instead. That would be much792 simpler, and a bit faster too.793 794 It's true that an in-memory Merkle tree could still provide the795 advantage of verification on every read rather than just on the796 first read. However, it would be inefficient because every time a797 hash page gets evicted (you can't pin the entire Merkle tree into798 memory, since it may be very large), in order to restore it you799 again need to hash everything below it in the tree. This again800 defeats most of the point of doing a Merkle tree-based hash, since801 a single block read could trigger re-hashing gigabytes of data.802 803:Q: But couldn't you store just the leaf nodes and compute the rest?804:A: See previous answer; this really just moves up one level, since805 one could alternatively interpret the data blocks as being the806 leaf nodes of the Merkle tree. It's true that the tree can be807 computed much faster if the leaf level is stored rather than just808 the data, but that's only because each level is less than 1% the809 size of the level below (assuming the recommended settings of810 SHA-256 and 4K blocks). For the exact same reason, by storing811 "just the leaf nodes" you'd already be storing over 99% of the812 tree, so you might as well simply store the whole tree.813 814:Q: Can the Merkle tree be built ahead of time, e.g. distributed as815 part of a package that is installed to many computers?816:A: This isn't currently supported. It was part of the original817 design, but was removed to simplify the kernel UAPI and because it818 wasn't a critical use case. Files are usually installed once and819 used many times, and cryptographic hashing is somewhat fast on820 most modern processors.821 822:Q: Why doesn't fs-verity support writes?823:A: Write support would be very difficult and would require a824 completely different design, so it's well outside the scope of825 fs-verity. Write support would require:826 827 - A way to maintain consistency between the data and hashes,828 including all levels of hashes, since corruption after a crash829 (especially of potentially the entire file!) is unacceptable.830 The main options for solving this are data journalling,831 copy-on-write, and log-structured volume. But it's very hard to832 retrofit existing filesystems with new consistency mechanisms.833 Data journalling is available on ext4, but is very slow.834 835 - Rebuilding the Merkle tree after every write, which would be836 extremely inefficient. Alternatively, a different authenticated837 dictionary structure such as an "authenticated skiplist" could838 be used. However, this would be far more complex.839 840 Compare it to dm-verity vs. dm-integrity. dm-verity is very841 simple: the kernel just verifies read-only data against a842 read-only Merkle tree. In contrast, dm-integrity supports writes843 but is slow, is much more complex, and doesn't actually support844 full-device authentication since it authenticates each sector845 independently, i.e. there is no "root hash". It doesn't really846 make sense for the same device-mapper target to support these two847 very different cases; the same applies to fs-verity.848 849:Q: Since verity files are immutable, why isn't the immutable bit set?850:A: The existing "immutable" bit (FS_IMMUTABLE_FL) already has a851 specific set of semantics which not only make the file contents852 read-only, but also prevent the file from being deleted, renamed,853 linked to, or having its owner or mode changed. These extra854 properties are unwanted for fs-verity, so reusing the immutable855 bit isn't appropriate.856 857:Q: Why does the API use ioctls instead of setxattr() and getxattr()?858:A: Abusing the xattr interface for basically arbitrary syscalls is859 heavily frowned upon by most of the Linux filesystem developers.860 An xattr should really just be an xattr on-disk, not an API to861 e.g. magically trigger construction of a Merkle tree.862 863:Q: Does fs-verity support remote filesystems?864:A: So far all filesystems that have implemented fs-verity support are865 local filesystems, but in principle any filesystem that can store866 per-file verity metadata can support fs-verity, regardless of867 whether it's local or remote. Some filesystems may have fewer868 options of where to store the verity metadata; one possibility is869 to store it past the end of the file and "hide" it from userspace870 by manipulating i_size. The data verification functions provided871 by ``fs/verity/`` also assume that the filesystem uses the Linux872 pagecache, but both local and remote filesystems normally do so.873 874:Q: Why is anything filesystem-specific at all? Shouldn't fs-verity875 be implemented entirely at the VFS level?876:A: There are many reasons why this is not possible or would be very877 difficult, including the following:878 879 - To prevent bypassing verification, folios must not be marked880 Uptodate until they've been verified. Currently, each881 filesystem is responsible for marking folios Uptodate via882 ``->readahead()``. Therefore, currently it's not possible for883 the VFS to do the verification on its own. Changing this would884 require significant changes to the VFS and all filesystems.885 886 - It would require defining a filesystem-independent way to store887 the verity metadata. Extended attributes don't work for this888 because (a) the Merkle tree may be gigabytes, but many889 filesystems assume that all xattrs fit into a single 4K890 filesystem block, and (b) ext4 and f2fs encryption doesn't891 encrypt xattrs, yet the Merkle tree *must* be encrypted when the892 file contents are, because it stores hashes of the plaintext893 file contents.894 895 So the verity metadata would have to be stored in an actual896 file. Using a separate file would be very ugly, since the897 metadata is fundamentally part of the file to be protected, and898 it could cause problems where users could delete the real file899 but not the metadata file or vice versa. On the other hand,900 having it be in the same file would break applications unless901 filesystems' notion of i_size were divorced from the VFS's,902 which would be complex and require changes to all filesystems.903 904 - It's desirable that FS_IOC_ENABLE_VERITY uses the filesystem's905 transaction mechanism so that either the file ends up with906 verity enabled, or no changes were made. Allowing intermediate907 states to occur after a crash may cause problems.908