TG:dZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m ZddlmZddlmZdd lmZej*eZd ZGd d eZGd dZGddeZGddeZy)a This module implements copying of images with bmap and provides the following API. 1. BmapCopy class - implements copying to any kind of file, be that a block device or a regular file. 2. BmapBdevCopy class - based on BmapCopy and specializes on copying to block devices. It does some more sanity checks and some block device performance tuning. The bmap file is an XML file which contains a list of mapped blocks of the image. Mapped blocks are the blocks which have disk sectors associated with them, as opposed to holes, which are blocks with no associated disk sectors. In other words, the image is considered to be a sparse file, and bmap basically contains a list of mapped blocks of this sparse file. The bmap additionally contains some useful information like block size (usually 4KiB), image size, mapped blocks count, etc. The bmap is used for copying the image to a block device or to a regular file. The idea is that we copy quickly with bmap because we copy only mapped blocks and ignore the holes, because they are useless. And if the image is generated properly (starting with a huge hole and writing all the data), it usually contains only little mapped blocks, comparing to the overall image size. And such an image compresses very well (because holes are read as all zeroes), so it is beneficial to distributor them as compressed files along with the bmap. Here is an example. Suppose you have a 4GiB image which contains only 100MiB of user data and you need to flash it to a slow USB stick. With bmap you end up copying only a little bit more than 100MiB of data from the image to the USB stick (namely, you copy only mapped blocks). This is a lot faster than copying all 4GiB of data. We say that it is a bit more than 100MiB because things like file-system meta-data (inode tables, superblocks, etc), partition table, etc also contribute to the mapped blocks and are also copied. N)reraise)queue)_thread)Optional) ElementTree) human_sizez2.0ceZdZdZy)Errorz A class for exceptions generated by the 'BmapCopy' module. We currently support only one type of exceptions, and we basically throw human-readable problem description in case of errors. N)__name__ __module__ __qualname____doc__6/usr/lib/python3.12/site-packages/bmaptool/BmapCopy.pyr r Os   rr cbeZdZdZej dZd dededdfdZdZ d Z d Z d Z y) SysfsChangea?Context manager for temporary sysfs changes Writes a temporary value to a sysfs file and restores the original value upon exit. Does a read access first to see if the requested value is already set. In that case, no write access is made to not cause unnecessary errors when running as a non-root user. Can deal with sysfs files that return their plain current value upon reading, as well as those that return all possible values with the current one in square brackets. In this case, all possible values are stored in the `options` attribute. z!([^\[\]]*)\[([^ \]]+)\]([^\[\]]*)path temp_valuereturnNcf||_||_||_d|_d|_g|_d|_y)avThe class constructor. The parameters are: path - The sysfs file to change temp_value - The value to set upon entering the context manager suppress_ioerrors - When True, an IOError during __enter__ is stored to the `error` attribute. When False, the exception is raised directly. FN)rrsuppress_ioerrors old_valuemodifiedoptionserror)selfrrrs r__init__zSysfsChange.__init__fs6 $!2  (, rc`t|jd5}|jj}ddd|jj }|rGdj |jddjd|_ |jdS|S#1swYoxYw)Nrrr ) openrreadstripre_current_selectionmatchjoingroupssplitrgroup)rfcontentsr*s r_readzSysfsChange._readvs $))S !Qvvx~~'H" ))//9 775<<>!A#67==cBDL;;q> !O" !s B$$B-c|t|jd5}|j|dddy#1swYyxYw)Nw)r&rwrite)rvaluer/s r_writezSysfsChange._writes) $))S !Q GGEN" ! !s2;c |j|_tjd|jd|jd|j|jk7rU tjd|jd|jd|j|jd|_ |S|S#t $r }|j r||_|cYd}~Sd}~wwxYw#t $r }|j r||_nYd}~|Sd}~wwxYw)Nzfound z to be ''setting  to 'T) r1r_logdebugrIOErrorrrrr6r)rexcs r __enter__zSysfsChange.__enter__s !ZZ\DN JJ {(4>>2B!D E >>T__ ,  Xdii[doo5FaHI DOO, $  t # %%      ))!$DJ  s=AB7!AC#7 C CC CC # D ,DD c (|jrM tjd|jd|jd|j |jyy#t $r-}td|jd|jd|d}~wwxYw)Nr9z back to 'r8zcannot restore r:': F)rr;r<rrr6r=r )rexc_type exc_valueexc_tbr>s r__exit__zSysfsChange.__exit__s == X Xdii[ 4>>:J!LM DNN+ Xodii[dnn=MSQTPUVWW XsA A B$(B  B)T) r r rrrecompiler)strr r1r6r?rErrrrrYsI&2::&JK-S-c-d- .rrc\eZdZdZddZdZdZdZdZdZ d Z d Z d Z d Z dd ZdZy)BmapCopya` This class implements the bmap-based copying functionality. To copy an image with bmap you should create an instance of this class, which requires the following: * full path or a file-like object of the image to copy * full path or a file object of the destination file copy the image to * full path or a file object of the bmap file (optional) * image size in bytes (optional) Although the main purpose of this class is to use bmap, the bmap is not required, and if it was not provided then the entire image will be copied to the destination file. When the bmap is provided, it is not necessary to specify image size, because the size is contained in the bmap. Otherwise, it is benefitial to specify the size because it enables extra sanity checks and makes it possible to provide the progress bar. When the image size is known either from the bmap or the caller specified it to the class constructor, all the image geometry description attributes ('blocks_cnt', etc) are initialized by the class constructor and available for the user. However, when the size is not known, some of the image geometry description attributes are not initialized by the class constructor. Instead, they are initialized only by the 'copy()' method. The 'copy()' method implements image copying. You may choose whether to verify the checksum while copying or not. Note, this is done only in case of bmap-based copying and only if bmap contains checksums (e.g., bmap version 1.0 did not have checksums support). You may choose whether to synchronize the destination file after writing or not. To explicitly synchronize it, use the 'sync()' method. This class supports all the bmap format versions up version 'SUPPORTED_BMAP_VERSION'. It is possible to have a simple progress indicator while copying the image. Use the 'set_progress_indicator()' method. You can copy only once with an instance of this class. This means that in order to copy the image for the second time, you have to create a new class instance. Ncd|_d|_d|_d|_d|_d|_d|_d|_d|_d|_ d|_ d|_ d|_ d|_ d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_|j1ddd|_||_|j6|_||_|j6|_t?j@|j:jC}tEjF|jH|_%d|_&d|_'d|_(d|_)tEjT|jHrLt?jV|jXdk(r*t?jZ|jXdk(rd|_.nd|_.|r)||_|j6|_/|jand|_d |_ d |_|r|jc||j|jz|_y) a  The class constructor. The parameters are: image - file-like object of the image which should be copied, should only support 'read()' and 'seek()' methods, and only seeking forward has to be supported. dest - file object of the destination file to copy the image to. bmap - file object of the bmap file to use for copying. image_size - size of the image in bytes. Nirr#FTrid)2_xml_dest_fsync_watermark _batch_blocks _batch_queue _batch_bytes_batch_queue_len bmap_versionbmap_version_majorbmap_version_minor block_size blocks_cnt mapped_cnt image_sizeimage_size_human mapped_sizemapped_size_humanmapped_percent_f_bmap _f_bmap_path_progress_started_progress_index_progress_time_progress_file_progress_formatset_progress_indicator _psplash_pipe_f_imagename _image_path_f_dest _dest_pathosfstatfilenostatS_ISREGst_mode_dest_is_regfile_cs_type_cs_len_cs_attrib_name_bmap_cs_attrib_nameS_ISCHRmajorst_rdevminor_dest_supports_fsync _bmap_path _parse_bmap_set_image_size)rimagedestbmaprZst_datas rr zBmapCopy.__init__s %)"! ' ! "&"& $!%"  !%#"" $ ##D$/!  :: ))((4<<..01 $ W__ =  #$(! LL ))Q.)Q.(-D %(,D % DL"iiDO    !"D "DO"%D     ,!..$//Arctjj|r:tjtj|j r||_ytjd|zy)a Set the psplash named pipe file path to be used when updating the progress - best effort. The 'path' argument is the named pipe used by the psplash process to get progress bar commands. When the path argument doesn't exist or is not a pipe, the function will ignore with a warning. This behavior is considered as such because the progress is considered a decoration functionality which might or might not be available even if requested. When used as a boot service, the unavailability of the psplash service (due to various reasons: no screen, racing issues etc.) should not break the writting process. This is why this implementation is done as a best effort. z;'%s' is not a pipe, so psplash progress will not be updatedN) rmrexistsrpS_ISFIFOrrrgr;warning)rrs rset_psplash_pipezBmapCopy.set_psplash_pipe,sJ 77>>$ DMM"''$-2G2G$H!%D  LLPSWW rc4||_|r||_yd|_y)a Setup the progress indicator which shows how much data has been copied in percent. The 'file_obj' argument is the console file object where the progress has to be printed to. Pass 'None' to disable the progress indicator. The 'format_string' argument is the format string for the progress indicator. It has to contain a single '%d' placeholder which will be substitutes with copied data in percent. z Copied %d%%N)rdre)rfile_obj format_strings rrfzBmapCopy.set_progress_indicatorCs' $1D !$1D !rc|j4|j|k7r%td||j|jfz||_t||_|j|jzdz |jz|_|j 4|j |_|j|_|j|_yy)zZ Set image size and initialize various other geometry-related attributes. NzBcannot set image size to %d bytes, it is known to be %d bytes (%s)r) rZr r[r rWrXrYr\r])rrZs rrzBmapCopy._set_image_sizeVs ?? &4??j+H#t0E0EFG  % *: 6??T__ R>BT\\>Q Jdll!:;;;t}}- $$..0  .??$5~G  /rc  tj|j|_|j}t|jjjd |_t|jj!d d d|_t|jj!d d d |_|j"tt&j!d d dkDrtd t&|j"fzt|j)d j*j-|_t|j)dj*j-|_t|j)dj*j-|_t|j)dj*j-|_t7|j4|_|j2|j.z|_t7|j:|_|j2dz|j0z |_|j4|j.zd z |j.z}|j0|k7r/td|j4|j0|j.fz|j"d kDs|j"d k(rL|j$dk(r=|j)dj*j-|_ d|_!d|_"n$|j$dk(rd|_ d|_!d|_"|j@rM tGtIjJ|j@jM|_'|jSyy#tj$r}|jj dd}t |jD];\}}||jddz k\s||jddzks2|d||fzz }=td|jd|d|d}~wwxYw#tP$r }td|j@d|d}~wwxYw)z\ Parse the bmap file and initialize corresponding class instance attributs. rrz Line %d: %szcannot parse the bmap file 'z%' which should be a proper XML file: z, the XML extract: Nversion.rzKonly bmap format version up to %d is supported, version %d is not supported BlockSize BlocksCountMappedBlocksCount ImageSizegY@z]Inconsistent bmap - image size does not match blocks count (%d bytes != %d blocks * %d bytes) ChecksumTypechksumBmapFileChecksumr#sha1 BmapFileSHA1z!cannot initialize hash function "z": )*rparser_rN ParseErrorseek enumeratepositionr r}rHgetrootattribgetrTintr-rUrVSUPPORTED_BMAP_VERSIONrrr(rWrXrYrZr r[r\r]r^rtrvrwlenrrrru ValueErrorr)rerr xml_extractnumlinexmlrXs rr~zBmapCopy._parse_bmaps  #))$,,7DIii 4 4 8 8 CD#&d&7&7&=&=c1&Ea&H"I"%d&7&7&=&=c1&Ea&H"I  " "S)?)E)Ec1)Ma)P%Q Q.)4+B+BCD chh{388>>@Achh}5::@@BCchh':;@@FFHIchh{388>>@A *4?? ;??T__<!+D,<,>K5??C6  R BF--QTU s<$N.!;Q.QA QQ%*QQ R Q;;Rcb|jr[||jksJtt||jz dz}tj d||j|fzntj d|z|j r|jrd|j zzdz}n}tjj}tjd}||jz |kry||_ d }d||jd zzdz}|xjd z c_ |jr|j jd nd |_ |j j||j j|jr|jrt t j"t j$z}t!j&t!j(|j|d5}|jdzdddyyy#1swYyxYw#YyxYw)z Print the progress indicator if the mapped area size is known and if the indicator has been enabled by assigning a console file object to the 'progress_file' attribute. rMz wrote %d blocks out of %d (%d%%)zwrote %d blocks  ) millisecondsN)-\|/rrzTr3z PROGRESS %d )rYrfloatr;r<rdredatetimenow timedeltarcrbrar4flushrgrmO_WRONLY O_NONBLOCKfdopenr&) rblocks_writtenpercentprogressr min_deltaprogress_wheelmodep_fos r_update_progresszBmapCopy._update_progresss ??!T__4 44504??BcIJG JJ2!4??G<=  JJ(>9 :   $"7"7'"AADH''++-$..C@ ,,,y8&)#!6.1E1E1I"JJTQ$$)$%%##)))4)-&    % %h /    % % '   $// {{R]]2YYrwwt'9'94@#F$JJ89GF#2 GF s+)AH*>HH*H'#H*'H**H.c#K|jsK|jrd|jdz dfyd} |||jzdz df||jz }'|j}|j d}|j dD]}|j j}|jddDcgc]}|j}}t|d}t|dkDr!t|d}||kDrtd|z|}|j|jvr|j|j} nd} ||| fycc}ww)a] This is a helper generator that parses the bmap XML file and for each block range in the XML file it yields ('first', 'last', 'chksum') tuples, where: * 'first' is the first block of the range; * 'last' is the last block of the range; * 'chksum' is the checksum of the range ('None' is used if it is missing). If there is no bmap file, the generator just yields a single range for entire image file. If the image size is unknown, the generator infinitely yields continuous ranges of size '_batch_blocks'. rrNBlockMapRangerzbad range (first > last): '%s')r_rXrPrNrfindallrr(r-rrr rvr) rfirstrxml_bmap xml_element blocks_rangexr-lastrs r_get_block_rangeszBmapCopy._get_block_rangess^||$//A-t44   %$*<*<"tjd|j|jjdy|r|rj!| t#| |jzdz |jz} tjd| |jj%fz|jjd||| zdz | f|ss|swj'|k7std|||j'||jfz |jjdy#t$r!} td|||j| fzd} ~ wwxYw#t($r2|jjd t+j,fYwxYw) aB This is generator which reads the image file in '_batch_blocks' chunks and yields ('type', 'start', 'end', 'buf) tuples, where: * 'start' is the starting block number of the batch; * 'end' is the last block of the batch; * 'buf' a buffer containing the batch data. zthe reader thread has startedz;error while reading blocks %d-%d of the image file '%s': %sNz#no more data to read from file '%s'rz&queueing %d blocks, queue length is %drangezUchecksum mismatch for blocks range %d-%d: calculated %s, should be %s (image file %s)r)r;r<rrrrtrhrrWrr'r=r rjrQputrrqsizer Exceptionsysexc_info) rverifyrrrriteratorstartendlengthbufrblockss r _get_datazBmapCopy._get_datahs6 23/ =)-)?)?)A%ff&{{4==9H ""54??#:;,,UD9,4(UC"mm00$//1IJ A4CSCS))--d3& ,!#h81<PFJJ@!4#4#4#:#:#<=> %%))7E56>A;Ms*ST3-56f););)=)GF $(:(:( :  !& &//335 ??t44 V T\\002DOOD %%))+E}qW$!8 Xa[(1+> %a UCs8e a4??BB BBs8sU{doo== == LL  edoo5 6))!Z$2L2L%LL!/JIIK  ""3'    ' ' ) cEkAo -N SX %M  ! !. 1IL   / T__ ,#$$OOOOOO       V T\\002DOOD J LL     IIK W VtPSTUU V> Bc4??C89 H VtPSTUU V  J4??CHI I Js`8K15L 8M M61 L:LL M&MM M3M..M36 N?NNc|jr0 tj|jj fyy#t $r+}t d|jd|jdd}~wwxYw)zv Synchronize the destination file to make sure all the data are actually written to the disk. zcannot synchronize 'rAr$N) r|rmfsyncrkrorr rlstrerror)rrs rrz BmapCopy.sync sb  $ $ ,,./0 % 6:oos||T s.> A2&A--A2NNTT)r r rrr rrfrrr~rrrrrrrrrrJrJsP-^QBf.2&;*@T)l7r4(lB0;$zfP rrJc,eZdZdZddZdfd ZxZS) BmapBdevCopya This class is a specialized version of 'BmapCopy' which copies the image to a block device. Unlike the base 'BmapCopy' class, this class does various optimizations specific to block devices, e.g., switching to the 'none' I/O scheduler. c Ftj|||||d|jz|_d|_d|_d|_|jr tj|jjdtj}tj|jjdtj||jkr?td|j$d|j&d |j d t)|d tj*|jjj,}d tj.|d tj0|d|_tj2j5|jdzs|jdz|_|jdz|_|jdz|_y#t$r+}td|j d|j"dd}~wwxYw)z The same as the constructor of the 'BmapCopy' base class, but adds useful guard-checks specific to block devices. i`Nrzcannot seed block device 'rAr$zthe image file 'z ' has size z' and it will not fit the block device 'z ' which has z capacityz/sys/dev/block/:rrz../zqueue/schedulerz bdi/max_ratio)rJr rWrO _sysfs_base_sysfs_scheduler_path_sysfs_max_ratio_pathrZrmlseekrkroSEEK_ENDSEEK_SETrr rlrrjr[r rnrzryr{rr)rrrrrZ bdev_sizerrzs rr zBmapBdevCopy.__init__$s $tT:>&5$//%I"%)"%)" ?? HHT\\%8%8%:Ar{{K ,,.2;;?4??*((--"9-   ((4<<..0199 HHW  HHW  ww~~d..89#//%7D %)%5%58I%I"%)%5%5%G"A  6 sA:G,, H 5&HH c >t|jd5}t|jd5}|jr#tj d|jd|jrAtj d|jd|jxsdd|jd|js |jr"tj d |jt|-||d d d d d d y #1swYxYw#1swYy xYw) a The same as in the base class but tunes the block device for better performance before starting writing. Additionally, it forces block device synchronization from time to time in order to make sure we do not get stuck in 'fsync()' for too long time. The problem is that the kernel synchronizes block devices when the file is closed. And the result is that if the user interrupts us while we are copying the data, the program will be blocked in 'close()' waiting for the block device synchronization, which may last minutes for slow USB stick. This is very bad user experience, and we work around this effect by synchronizing from time to time. 1nonezsfailed to disable excessive buffering, expect worse system responsiveness (reason: cannot set max. I/O ratio to 1: )zZfailed to enable I/O optimization, expect suboptimal speed (reason: cannot switch to the 'z' I/O scheduler: 'zunknown schedulerz ' in use. aSYou may want to set these I/O optimizations through a udev rule like this: #/etc/udev/rules.d/60-bmaptool-optimizations.rules SUBSYSTEM!="block", GOTO="bmaptool_optimizations_end" ACTION!="add|change", GOTO="bmaptool_optimizations_end" ACTION=="add", SUBSYSTEMS=="usb", ATTRS{idVendor}=="xxxx", ATTRS{idProduct}=="xxxx", TAG+="uaccess" SUBSYSTEMS=="usb", ENV{DEVTYPE}!="partition", ATTRS{idVendor}=="xxxx", ATTRS{idProduct}=="xxxx", ATTR{bdi/min_ratio}="0", ATTR{bdi/max_ratio}="1", ATTR{queue/scheduler}="none" LABEL="bmaptool_optimizations_end" For attributes to match, try udevadm info -a N) rr r rr;rinforrrlsuperr)rrr max_ratio_chg scheduler_chg __class__s rrzBmapBdevCopy.copyZs033S 9]K  & &M "" ,,9,?,?+@C "" %0012%//F3FGz$**+1 .""m&9&9 ' (,&7!9( GLv &MM 9 9M M 9 9s#DCD6DD DDrr)r r rrr r __classcell__)rs@rrrs4Hl>'>'rr)rrmrFrprrloggingrsixr six.movesrrrrtypingr xml.etreer BmapHelpersr getLoggerr r;rrr robjectrJrrrrr#s" T $'!#w" I MM`p vp f|'8|'r