566 lines · plain
1========================================2zram: Compressed RAM-based block devices3========================================4 5Introduction6============7 8The zram module creates RAM-based block devices named /dev/zram<id>9(<id> = 0, 1, ...). Pages written to these disks are compressed and stored10in memory itself. These disks allow very fast I/O and compression provides11good amounts of memory savings. Some of the use cases include /tmp storage,12use as swap disks, various caches under /var and maybe many more. :)13 14Statistics for individual zram devices are exported through sysfs nodes at15/sys/block/zram<id>/16 17Usage18=====19 20There are several ways to configure and manage zram device(-s):21 22a) using zram and zram_control sysfs attributes23b) using zramctl utility, provided by util-linux (util-linux@vger.kernel.org).24 25In this document we will describe only 'manual' zram configuration steps,26IOW, zram and zram_control sysfs attributes.27 28In order to get a better idea about zramctl please consult util-linux29documentation, zramctl man-page or `zramctl --help`. Please be informed30that zram maintainers do not develop/maintain util-linux or zramctl, should31you have any questions please contact util-linux@vger.kernel.org32 33Following shows a typical sequence of steps for using zram.34 35WARNING36=======37 38For the sake of simplicity we skip error checking parts in most of the39examples below. However, it is your sole responsibility to handle errors.40 41zram sysfs attributes always return negative values in case of errors.42The list of possible return codes:43 44======== =============================================================45-EBUSY an attempt to modify an attribute that cannot be changed once46 the device has been initialised. Please reset device first.47-ENOMEM zram was not able to allocate enough memory to fulfil your48 needs.49-EINVAL invalid input has been provided.50======== =============================================================51 52If you use 'echo', the returned value is set by the 'echo' utility,53and, in general case, something like::54 55 echo 3 > /sys/block/zram0/max_comp_streams56 if [ $? -ne 0 ]; then57 handle_error58 fi59 60should suffice.61 621) Load Module63==============64 65::66 67 modprobe zram num_devices=468 69This creates 4 devices: /dev/zram{0,1,2,3}70 71num_devices parameter is optional and tells zram how many devices should be72pre-created. Default: 1.73 742) Set max number of compression streams75========================================76 77Regardless of the value passed to this attribute, ZRAM will always78allocate multiple compression streams - one per online CPU - thus79allowing several concurrent compression operations. The number of80allocated compression streams goes down when some of the CPUs81become offline. There is no single-compression-stream mode anymore,82unless you are running a UP system or have only 1 CPU online.83 84To find out how many streams are currently available::85 86 cat /sys/block/zram0/max_comp_streams87 883) Select compression algorithm89===============================90 91Using comp_algorithm device attribute one can see available and92currently selected (shown in square brackets) compression algorithms,93or change the selected compression algorithm (once the device is initialised94there is no way to change compression algorithm).95 96Examples::97 98 #show supported compression algorithms99 cat /sys/block/zram0/comp_algorithm100 lzo [lz4]101 102 #select lzo compression algorithm103 echo lzo > /sys/block/zram0/comp_algorithm104 105For the time being, the `comp_algorithm` content shows only compression106algorithms that are supported by zram.107 1084) Set compression algorithm parameters: Optional109=================================================110 111Compression algorithms may support specific parameters which can be112tweaked for particular dataset. ZRAM has an `algorithm_params` device113attribute which provides a per-algorithm params configuration.114 115For example, several compression algorithms support `level` parameter.116In addition, certain compression algorithms support pre-trained dictionaries,117which significantly change algorithms' characteristics. In order to configure118compression algorithm to use external pre-trained dictionary, pass full119path to the `dict` along with other parameters::120 121 #pass path to pre-trained zstd dictionary122 echo "algo=zstd dict=/etc/dictioary" > /sys/block/zram0/algorithm_params123 124 #same, but using algorithm priority125 echo "priority=1 dict=/etc/dictioary" > \126 /sys/block/zram0/algorithm_params127 128 #pass path to pre-trained zstd dictionary and compression level129 echo "algo=zstd level=8 dict=/etc/dictioary" > \130 /sys/block/zram0/algorithm_params131 132Parameters are algorithm specific: not all algorithms support pre-trained133dictionaries, not all algorithms support `level`. Furthermore, for certain134algorithms `level` controls the compression level (the higher the value the135better the compression ratio, it even can take negatives values for some136algorithms), for other algorithms `level` is acceleration level (the higher137the value the lower the compression ratio).138 1395) Set Disksize140===============141 142Set disk size by writing the value to sysfs node 'disksize'.143The value can be either in bytes or you can use mem suffixes.144Examples::145 146 # Initialize /dev/zram0 with 50MB disksize147 echo $((50*1024*1024)) > /sys/block/zram0/disksize148 149 # Using mem suffixes150 echo 256K > /sys/block/zram0/disksize151 echo 512M > /sys/block/zram0/disksize152 echo 1G > /sys/block/zram0/disksize153 154Note:155There is little point creating a zram of greater than twice the size of memory156since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of the157size of the disk when not in use so a huge zram is wasteful.158 1596) Set memory limit: Optional160=============================161 162Set memory limit by writing the value to sysfs node 'mem_limit'.163The value can be either in bytes or you can use mem suffixes.164In addition, you could change the value in runtime.165Examples::166 167 # limit /dev/zram0 with 50MB memory168 echo $((50*1024*1024)) > /sys/block/zram0/mem_limit169 170 # Using mem suffixes171 echo 256K > /sys/block/zram0/mem_limit172 echo 512M > /sys/block/zram0/mem_limit173 echo 1G > /sys/block/zram0/mem_limit174 175 # To disable memory limit176 echo 0 > /sys/block/zram0/mem_limit177 1787) Activate179===========180 181::182 183 mkswap /dev/zram0184 swapon /dev/zram0185 186 mkfs.ext4 /dev/zram1187 mount /dev/zram1 /tmp188 1898) Add/remove zram devices190==========================191 192zram provides a control interface, which enables dynamic (on-demand) device193addition and removal.194 195In order to add a new /dev/zramX device, perform a read operation on the hot_add196attribute. This will return either the new device's device id (meaning that you197can use /dev/zram<id>) or an error code.198 199Example::200 201 cat /sys/class/zram-control/hot_add202 1203 204To remove the existing /dev/zramX device (where X is a device id)205execute::206 207 echo X > /sys/class/zram-control/hot_remove208 2099) Stats210========211 212Per-device statistics are exported as various nodes under /sys/block/zram<id>/213 214A brief description of exported device attributes follows. For more details215please read Documentation/ABI/testing/sysfs-block-zram.216 217====================== ====== ===============================================218Name access description219====================== ====== ===============================================220disksize RW show and set the device's disk size221initstate RO shows the initialization state of the device222reset WO trigger device reset223mem_used_max WO reset the `mem_used_max` counter (see later)224mem_limit WO specifies the maximum amount of memory ZRAM can225 use to store the compressed data226writeback_limit WO specifies the maximum amount of write IO zram227 can write out to backing device as 4KB unit228writeback_limit_enable RW show and set writeback_limit feature229max_comp_streams RW the number of possible concurrent compress230 operations231comp_algorithm RW show and change the compression algorithm232algorithm_params WO setup compression algorithm parameters233compact WO trigger memory compaction234debug_stat RO this file is used for zram debugging purposes235backing_dev RW set up backend storage for zram to write out236idle WO mark allocated slot as idle237====================== ====== ===============================================238 239 240User space is advised to use the following files to read the device statistics.241 242File /sys/block/zram<id>/stat243 244Represents block layer statistics. Read Documentation/block/stat.rst for245details.246 247File /sys/block/zram<id>/io_stat248 249The stat file represents device's I/O statistics not accounted by block250layer and, thus, not available in zram<id>/stat file. It consists of a251single line of text and contains the following stats separated by252whitespace:253 254 ============= =============================================================255 failed_reads The number of failed reads256 failed_writes The number of failed writes257 invalid_io The number of non-page-size-aligned I/O requests258 notify_free Depending on device usage scenario it may account259 260 a) the number of pages freed because of swap slot free261 notifications262 b) the number of pages freed because of263 REQ_OP_DISCARD requests sent by bio. The former ones are264 sent to a swap block device when a swap slot is freed,265 which implies that this disk is being used as a swap disk.266 267 The latter ones are sent by filesystem mounted with268 discard option, whenever some data blocks are getting269 discarded.270 ============= =============================================================271 272File /sys/block/zram<id>/mm_stat273 274The mm_stat file represents the device's mm statistics. It consists of a single275line of text and contains the following stats separated by whitespace:276 277 ================ =============================================================278 orig_data_size uncompressed size of data stored in this disk.279 Unit: bytes280 compr_data_size compressed size of data stored in this disk281 mem_used_total the amount of memory allocated for this disk. This282 includes allocator fragmentation and metadata overhead,283 allocated for this disk. So, allocator space efficiency284 can be calculated using compr_data_size and this statistic.285 Unit: bytes286 mem_limit the maximum amount of memory ZRAM can use to store287 the compressed data288 mem_used_max the maximum amount of memory zram has consumed to289 store the data290 same_pages the number of same element filled pages written to this disk.291 No memory is allocated for such pages.292 pages_compacted the number of pages freed during compaction293 huge_pages the number of incompressible pages294 huge_pages_since the number of incompressible pages since zram set up295 ================ =============================================================296 297File /sys/block/zram<id>/bd_stat298 299The bd_stat file represents a device's backing device statistics. It consists of300a single line of text and contains the following stats separated by whitespace:301 302 ============== =============================================================303 bd_count size of data written in backing device.304 Unit: 4K bytes305 bd_reads the number of reads from backing device306 Unit: 4K bytes307 bd_writes the number of writes to backing device308 Unit: 4K bytes309 ============== =============================================================310 31110) Deactivate312==============313 314::315 316 swapoff /dev/zram0317 umount /dev/zram1318 31911) Reset320=========321 322 Write any positive value to 'reset' sysfs node::323 324 echo 1 > /sys/block/zram0/reset325 echo 1 > /sys/block/zram1/reset326 327 This frees all the memory allocated for the given device and328 resets the disksize to zero. You must set the disksize again329 before reusing the device.330 331Optional Feature332================333 334writeback335---------336 337With CONFIG_ZRAM_WRITEBACK, zram can write idle/incompressible page338to backing storage rather than keeping it in memory.339To use the feature, admin should set up backing device via::340 341 echo /dev/sda5 > /sys/block/zramX/backing_dev342 343before disksize setting. It supports only partitions at this moment.344If admin wants to use incompressible page writeback, they could do it via::345 346 echo huge > /sys/block/zramX/writeback347 348To use idle page writeback, first, user need to declare zram pages349as idle::350 351 echo all > /sys/block/zramX/idle352 353From now on, any pages on zram are idle pages. The idle mark354will be removed until someone requests access of the block.355IOW, unless there is access request, those pages are still idle pages.356Additionally, when CONFIG_ZRAM_TRACK_ENTRY_ACTIME is enabled pages can be357marked as idle based on how long (in seconds) it's been since they were358last accessed::359 360 echo 86400 > /sys/block/zramX/idle361 362In this example all pages which haven't been accessed in more than 86400363seconds (one day) will be marked idle.364 365Admin can request writeback of those idle pages at right timing via::366 367 echo idle > /sys/block/zramX/writeback368 369With the command, zram will writeback idle pages from memory to the storage.370 371Additionally, if a user choose to writeback only huge and idle pages372this can be accomplished with::373 374 echo huge_idle > /sys/block/zramX/writeback375 376If a user chooses to writeback only incompressible pages (pages that none of377algorithms can compress) this can be accomplished with::378 379 echo incompressible > /sys/block/zramX/writeback380 381If an admin wants to write a specific page in zram device to the backing device,382they could write a page index into the interface::383 384 echo "page_index=1251" > /sys/block/zramX/writeback385 386If there are lots of write IO with flash device, potentially, it has387flash wearout problem so that admin needs to design write limitation388to guarantee storage health for entire product life.389 390To overcome the concern, zram supports "writeback_limit" feature.391The "writeback_limit_enable"'s default value is 0 so that it doesn't limit392any writeback. IOW, if admin wants to apply writeback budget, they should393enable writeback_limit_enable via::394 395 $ echo 1 > /sys/block/zramX/writeback_limit_enable396 397Once writeback_limit_enable is set, zram doesn't allow any writeback398until admin sets the budget via /sys/block/zramX/writeback_limit.399 400(If admin doesn't enable writeback_limit_enable, writeback_limit's value401assigned via /sys/block/zramX/writeback_limit is meaningless.)402 403If admin wants to limit writeback as per-day 400M, they could do it404like below::405 406 $ MB_SHIFT=20407 $ 4K_SHIFT=12408 $ echo $((400<<MB_SHIFT>>4K_SHIFT)) > \409 /sys/block/zram0/writeback_limit.410 $ echo 1 > /sys/block/zram0/writeback_limit_enable411 412If admins want to allow further write again once the budget is exhausted,413they could do it like below::414 415 $ echo $((400<<MB_SHIFT>>4K_SHIFT)) > \416 /sys/block/zram0/writeback_limit417 418If an admin wants to see the remaining writeback budget since last set::419 420 $ cat /sys/block/zramX/writeback_limit421 422If an admin wants to disable writeback limit, they could do::423 424 $ echo 0 > /sys/block/zramX/writeback_limit_enable425 426The writeback_limit count will reset whenever you reset zram (e.g.,427system reboot, echo 1 > /sys/block/zramX/reset) so keeping how many of428writeback happened until you reset the zram to allocate extra writeback429budget in next setting is user's job.430 431If admin wants to measure writeback count in a certain period, they could432know it via /sys/block/zram0/bd_stat's 3rd column.433 434recompression435-------------436 437With CONFIG_ZRAM_MULTI_COMP, zram can recompress pages using alternative438(secondary) compression algorithms. The basic idea is that alternative439compression algorithm can provide better compression ratio at a price of440(potentially) slower compression/decompression speeds. Alternative compression441algorithm can, for example, be more successful compressing huge pages (those442that default algorithm failed to compress). Another application is idle pages443recompression - pages that are cold and sit in the memory can be recompressed444using more effective algorithm and, hence, reduce zsmalloc memory usage.445 446With CONFIG_ZRAM_MULTI_COMP, zram supports up to 4 compression algorithms:447one primary and up to 3 secondary ones. Primary zram compressor is explained448in "3) Select compression algorithm", secondary algorithms are configured449using recomp_algorithm device attribute.450 451Example:::452 453 #show supported recompression algorithms454 cat /sys/block/zramX/recomp_algorithm455 #1: lzo lzo-rle lz4 lz4hc [zstd]456 #2: lzo lzo-rle lz4 [lz4hc] zstd457 458Alternative compression algorithms are sorted by priority. In the example459above, zstd is used as the first alternative algorithm, which has priority460of 1, while lz4hc is configured as a compression algorithm with priority 2.461Alternative compression algorithm's priority is provided during algorithms462configuration:::463 464 #select zstd recompression algorithm, priority 1465 echo "algo=zstd priority=1" > /sys/block/zramX/recomp_algorithm466 467 #select deflate recompression algorithm, priority 2468 echo "algo=deflate priority=2" > /sys/block/zramX/recomp_algorithm469 470Another device attribute that CONFIG_ZRAM_MULTI_COMP enables is recompress,471which controls recompression.472 473Examples:::474 475 #IDLE pages recompression is activated by `idle` mode476 echo "type=idle" > /sys/block/zramX/recompress477 478 #HUGE pages recompression is activated by `huge` mode479 echo "type=huge" > /sys/block/zram0/recompress480 481 #HUGE_IDLE pages recompression is activated by `huge_idle` mode482 echo "type=huge_idle" > /sys/block/zramX/recompress483 484The number of idle pages can be significant, so user-space can pass a size485threshold (in bytes) to the recompress knob: zram will recompress only pages486of equal or greater size:::487 488 #recompress all pages larger than 3000 bytes489 echo "threshold=3000" > /sys/block/zramX/recompress490 491 #recompress idle pages larger than 2000 bytes492 echo "type=idle threshold=2000" > /sys/block/zramX/recompress493 494It is also possible to limit the number of pages zram re-compression will495attempt to recompress:::496 497 echo "type=huge_idle max_pages=42" > /sys/block/zramX/recompress498 499Recompression of idle pages requires memory tracking.500 501During re-compression for every page, that matches re-compression criteria,502ZRAM iterates the list of registered alternative compression algorithms in503order of their priorities. ZRAM stops either when re-compression was504successful (re-compressed object is smaller in size than the original one)505and matches re-compression criteria (e.g. size threshold) or when there are506no secondary algorithms left to try. If none of the secondary algorithms can507successfully re-compressed the page such a page is marked as incompressible,508so ZRAM will not attempt to re-compress it in the future.509 510This re-compression behaviour, when it iterates through the list of511registered compression algorithms, increases our chances of finding the512algorithm that successfully compresses a particular page. Sometimes, however,513it is convenient (and sometimes even necessary) to limit recompression to514only one particular algorithm so that it will not try any other algorithms.515This can be achieved by providing a `algo` or `priority` parameter:::516 517 #use zstd algorithm only (if registered)518 echo "type=huge algo=zstd" > /sys/block/zramX/recompress519 520 #use zstd algorithm only (if zstd was registered under priority 1)521 echo "type=huge priority=1" > /sys/block/zramX/recompress522 523memory tracking524===============525 526With CONFIG_ZRAM_MEMORY_TRACKING, user can know information of the527zram block. It could be useful to catch cold or incompressible528pages of the process with*pagemap.529 530If you enable the feature, you could see block state via531/sys/kernel/debug/zram/zram0/block_state". The output is as follows::532 533 300 75.033841 .wh...534 301 63.806904 s.....535 302 63.806919 ..hi..536 303 62.801919 ....r.537 304 146.781902 ..hi.n538 539First column540 zram's block index.541Second column542 access time since the system was booted543Third column544 state of the block:545 546 s:547 same page548 w:549 written page to backing store550 h:551 huge page552 i:553 idle page554 r:555 recompressed page (secondary compression algorithm)556 n:557 none (including secondary) of algorithms could compress it558 559First line of above example says 300th block is accessed at 75.033841sec560and the block's state is huge so it is written back to the backing561storage. It's a debugging feature so anyone shouldn't rely on it to work562properly.563 564Nitin Gupta565ngupta@vflare.org566