. "$BST_LIB" # LZO is a real-time compression library: nine algorithm families (lzo1, lzo1a, # lzo1b, lzo1c, lzo1f, lzo1x, lzo1y, lzo1z, lzo2a) over one common core of # init/version/checksum/string entry points. It ships NO CLI of its own -- the # lzop(1) command line is a SEPARATE poky recipe, covered by 57-lzop.test -- so, # exactly as for zlib (57-zlib.test), the library is exercised here through an # in-image consumer that statically links it. That consumer is mtd-utils: lzo is # mkfs.ubifs's default compressor and one of mkfs.jffs2's three, and it is the # only thing on this image that links liblzo2 at all. # # FOUR SHAPE FACTS decide how this file is written. Each was measured against # the shipped payload before it was asserted, and each would produce a red on a # perfectly good port if it were assumed the other way. # # 1. THE HEADERS LIVE IN /usr/include/lzo/, NOT /usr/include/. All thirteen # are under the subdirectory, which is upstream's own layout and is why the # pkg-config file says `Cflags: -I${includedir}/lzo` rather than plain # -I${includedir}. A suite that looked for /usr/include/lzo1x.h would report # a correct port as missing, so the absence of the top-level copies is # asserted deliberately below rather than left unstated. # # 2. THE INSTALLED HEADERS ARE VERBATIM UPSTREAM -- there is no configure # substitution in any of them. That is the opposite of gmp, whose gmp-32.h # carries a configure-written GMP_LIMB_BITS that serves as its arm witness. # lzo has no such field, so NOTHING in the header text can distinguish this # wasm32 build from a native one, and this suite does not pretend otherwise: # its arm witness is fact 3, in the archive. # # 3. THE ARCHIVE MEMBERS ARE LLVM IR BITCODE, NOT NATIVE OBJECTS. This # toolchain defers codegen to link time, so every member of liblzo2.a opens # with the bitcode magic BC C0 DE (42 43 c0 de) -- not the ELF magic a # native x86_64 lzo would carry, and not the \0asm of a finished wasm # module either. Measured across gmp, popt and libyaml as well, so it is a # property of the arm and not of this recipe. This is the ONE assertion in # this file that a native lzo fails, which makes it the arm witness fact 2 # says the headers cannot provide. # # 4. THE ARCHIVE IS CHECKED THROUGH ITS SYMBOL INDEX, NOT ITS WHOLE BODY. # liblzo2.a is ~1 MB but its ar symbol index -- member "/", the map of # exported symbol to member offset -- is only 2716 bytes. Reading a 16 KB # window gives the same evidence for 1/64th of the target-side temp space, # and is the more precise place to look: the index lists what the archive # EXPORTS, not merely what bytes appear somewhere inside it. Member NAMES # are deliberately not asserted -- all 67 are shorter than 15 characters, so # they live inline in their own member headers scattered through the body # rather than in the index window (the trap that would have cost gmp's suite # sixteen false reds). # # PROCESS COUNT, NOT WORK, is what costs a suite its 60 s budget on this runtime # (the lesson 56-lz4.test learned the hard way and 96-mtd-utils-jffs2 records). # So each header is read into a shell variable ONCE and every declaration below # is a `case` match against it -- roughly a hundred assertions for ten forks # instead of a hundred. Line-start anchoring is done by matching a leading # newline, which is what makes a `case` match as precise as `grep -E "^name"` # without paying a process for it. inc=/usr/include/lzo pc=/usr/lib/pkgconfig/lzo2.pc lib=/usr/lib/liblzo2.a NL=' ' # ok iff appears anywhere in . has() { case "$3" in *"$2"*) bst_ok "$1" ;; *) bst_notok "$1" "want '$2'" ;; esac; } hasnt() { case "$3" in *"$2"*) bst_notok "$1" "unexpectedly present: '$2'" ;; *) bst_ok "$1" ;; esac; } # ok iff begins a line of -- i.e. it is a DECLARATION and not # a mention of the same name inside a comment or an argument list. decl() { case "$3" in *"$NL$2"*) bst_ok "$1" ;; *) bst_notok "$1" "no line-start declaration of '$2'" ;; esac; } size() { wc -c < "$1" 2>/dev/null | tr -d ' '; } # ------------------------------------------------------------------ payload --- # The three shipped pieces. Everything below reads one of them, so a failure # here explains every later red in this file at a glance. check lzo/inc-dir-present test -d "$inc" check lzo/lib-present test -f "$lib" check lzo/pc-present test -f "$pc" # All thirteen headers, by name. lzodefs.h is the 127 KB internal configuration # header -- it is not an API surface but it IS part of the shipped -dev package, # and lzo1x.h includes lzoconf.h which includes it, so a missing one breaks # every consumer at #include time rather than at link time. for h in lzo1.h lzo1a.h lzo1b.h lzo1c.h lzo1f.h lzo1x.h lzo1y.h lzo1z.h \ lzo2a.h lzo_asm.h lzoconf.h lzodefs.h lzoutil.h; do check "lzo/hdr-$h" test -s "$inc/$h" done # Shape fact 1, asserted as an absence. If a future recipe change flattened the # headers into /usr/include the pkg-config Cflags above would silently stop # matching the layout, and every consumer's #include would break. check lzo/hdr-not-toplevel-conf sh -c '[ ! -e /usr/include/lzoconf.h ]' check lzo/hdr-not-toplevel-1x sh -c '[ ! -e /usr/include/lzo1x.h ]' # Read once, match many -- see the process-count note in the header. HC=$(cat "$inc/lzoconf.h" 2>/dev/null) H1=$(cat "$inc/lzo1.h" 2>/dev/null) HA=$(cat "$inc/lzo1a.h" 2>/dev/null) HB=$(cat "$inc/lzo1b.h" 2>/dev/null) HCC=$(cat "$inc/lzo1c.h" 2>/dev/null) HF=$(cat "$inc/lzo1f.h" 2>/dev/null) HX=$(cat "$inc/lzo1x.h" 2>/dev/null) HY=$(cat "$inc/lzo1y.h" 2>/dev/null) HZ=$(cat "$inc/lzo1z.h" 2>/dev/null) H2A=$(cat "$inc/lzo2a.h" 2>/dev/null) # ------------------------------------------------------------------ version --- # Extracted and compared rather than substring-matched, because the macro and # its value are separated by padding. NOTE the STRING and DATE values keep their # surrounding quotes -- they are C string literals in the header text, so an # assertion against a bare 2.10 would go red on a correct header. check_eq lzo/version-macro 0x20a0 "$(awk '/^#define LZO_VERSION[ \t]/{print $3}' "$inc/lzoconf.h")" check_eq lzo/version-string '"2.10"' "$(awk '/^#define LZO_VERSION_STRING[ \t]/{print $3}' "$inc/lzoconf.h")" check_eq lzo/version-date '"Mar 01 2017"' "$(awk '/^#define LZO_VERSION_DATE[ \t]/{print $3, $4, $5}' "$inc/lzoconf.h")" # 0x20a0 is 2.10 encoded as upstream's packed major/minor/patch, so this pins # the header and the pkg-config Version to the same release. has lzo/version-comment "/* 2.10 */" "$HC" # ------------------------------------------------------------- core: lzoconf --- # The entry points every algorithm family shares. lzo_init() is a MACRO, not a # function -- it expands to __lzo_init_v2 passing the caller's sizeof()s so the # library can refuse a header/binary mismatch at run time. Asserting the macro # form and the underlying symbol separately keeps that distinction explicit. has lzo/api-init-macro "#define lzo_init() __lzo_init_v2(LZO_VERSION," "$HC" has lzo/api-init-v2 "LZO_EXTERN(int) __lzo_init_v2(" "$HC" has lzo/api-version "LZO_EXTERN(unsigned) lzo_version(void);" "$HC" has lzo/api-version-str "LZO_EXTERN(const char *) lzo_version_string(void);" "$HC" has lzo/api-version-dat "LZO_EXTERN(const char *) lzo_version_date(void);" "$HC" has lzo/api-config-chk "LZO_EXTERN(int) _lzo_config_check(void);" "$HC" has lzo/api-align-gap "LZO_EXTERN(unsigned) __lzo_align_gap(" "$HC" # The string and checksum helpers are declared with the type on one line and the # name indented on the next, so they are matched in that form. for f in lzo_memcmp lzo_memcpy lzo_memmove lzo_memset; do decl "lzo/api-$f" " $f(" "$HC" done for f in lzo_adler32 lzo_crc32 lzo_get_crc32_table; do decl "lzo/api-$f" " $f(" "$HC" done # ------------------------------------------------------------- error codes --- # The full LZO_E_* contract. A caller distinguishes a corrupt stream # (INPUT_OVERRUN, LOOKBEHIND_OVERRUN, EOF_NOT_FOUND) from a caller mistake # (INVALID_ARGUMENT, OUTPUT_OVERRUN) purely by these values, so their exact # numbering is API, not decoration. has lzo/err-ok "#define LZO_E_OK 0" "$HC" has lzo/err-error "#define LZO_E_ERROR (-1)" "$HC" has lzo/err-out-of-memory "#define LZO_E_OUT_OF_MEMORY (-2)" "$HC" has lzo/err-not-compress "#define LZO_E_NOT_COMPRESSIBLE (-3)" "$HC" has lzo/err-input-overrun "#define LZO_E_INPUT_OVERRUN (-4)" "$HC" has lzo/err-output-overrun "#define LZO_E_OUTPUT_OVERRUN (-5)" "$HC" has lzo/err-lookbehind "#define LZO_E_LOOKBEHIND_OVERRUN (-6)" "$HC" has lzo/err-eof-not-found "#define LZO_E_EOF_NOT_FOUND (-7)" "$HC" has lzo/err-not-consumed "#define LZO_E_INPUT_NOT_CONSUMED (-8)" "$HC" has lzo/err-not-yet-impl "#define LZO_E_NOT_YET_IMPLEMENTED (-9)" "$HC" has lzo/err-invalid-arg "#define LZO_E_INVALID_ARGUMENT (-10)" "$HC" has lzo/err-invalid-align "#define LZO_E_INVALID_ALIGNMENT (-11)" "$HC" has lzo/err-out-not-cons "#define LZO_E_OUTPUT_NOT_CONSUMED (-12)" "$HC" has lzo/err-internal "#define LZO_E_INTERNAL_ERROR (-99)" "$HC" # ------------------------------------------------------------------- types --- # The public vocabulary every entry point is written in. lzo_bytep/lzo_uint are # the pair that appears in every compress/decompress signature; lzo_callback_t # is the progress/allocation hook that lets LZO be embedded in an # allocator-constrained host, which is directly relevant on this target. has lzo/type-callback "typedef struct lzo_callback_t lzo_callback_t;" "$HC" has lzo/type-alloc-fn "typedef lzo_voidp (__LZO_CDECL *lzo_alloc_func_t)" "$HC" has lzo/type-align-t "} lzo_align_t;" "$HC" # The INTEGER types are real typedefs, selected by a preprocessor ladder over # the target's word sizes -- hence several spellings each, so only the typedef'd # name is matched. has lzo/type-uint "lzo_uint;" "$HC" has lzo/type-int "lzo_int;" "$HC" # The POINTER vocabulary, however, is #defines rather than typedefs, and the # header states the reason in place: "no typedef here because of const-pointer # issues". Asserting these in the obvious typedef form is a red on a perfectly # correct header -- measured, not hypothetical: this suite did exactly that and # the host dry run caught it before the port shipped. Each carries the # __LZO_MMODEL qualifier, LZO's 16-bit-memory-model legacy, which is why the # macro form is load-bearing rather than a style choice. has lzo/type-ptr-not-typedef "/* no typedef here because of const-pointer issues */" "$HC" has lzo/type-mmodel "#define __LZO_MMODEL" "$HC" for m in lzo_bytep lzo_charp lzo_voidp lzo_uintp lzo_voidpp lzo_bytepp; do decl "lzo/type-$m" "#define $m " "$HC" done # ---------------------------------------------------- family 1: lzo1x (main) --- # lzo1x is the family everything else is measured against and the one mkfs.jffs2 # and mkfs.ubifs actually call. _1_ is the standard compressor, _999_ the # maximum-ratio one, and decompress_safe the bounds-checked reader that any code # handling untrusted input must use. for f in lzo1x_decompress lzo1x_decompress_safe lzo1x_1_compress \ lzo1x_1_11_compress lzo1x_1_12_compress lzo1x_1_15_compress \ lzo1x_999_compress lzo1x_999_compress_dict lzo1x_999_compress_level \ lzo1x_decompress_dict_safe lzo1x_optimize; do decl "lzo/x-$f" "$f" "$HX" done has lzo/x-mem-compress "#define LZO1X_1_MEM_COMPRESS" "$HX" has lzo/x-mem-999 "#define LZO1X_999_MEM_COMPRESS" "$HX" has lzo/x-mem-decompress "#define LZO1X_MEM_DECOMPRESS (0)" "$HX" # LZO1X_MEM_DECOMPRESS being ZERO is the property that makes LZO usable in a # kernel or a bootloader -- decompression needs no work memory at all. It is # asserted by value for that reason, not merely by name. # ------------------------------------------------- family 2: lzo1y and lzo1z --- # lzo1y is lzo1x retuned; lzo1z is the large-window variant. Both are # decompress+999 only -- neither ships a _1_ compressor -- so asserting one here # would be a red on a correct header. for f in lzo1y_decompress lzo1y_decompress_safe lzo1y_1_compress \ lzo1y_999_compress lzo1y_999_compress_dict lzo1y_999_compress_level \ lzo1y_decompress_dict_safe lzo1y_optimize; do decl "lzo/y-$f" "$f" "$HY" done for f in lzo1z_decompress lzo1z_decompress_safe lzo1z_999_compress \ lzo1z_999_compress_dict lzo1z_999_compress_level \ lzo1z_decompress_dict_safe; do decl "lzo/z-$f" "$f" "$HZ" done hasnt lzo/z-no-1-compress "${NL}lzo1z_1_compress" "$HZ" # --------------------------------------------- family 3: lzo1b and lzo1c (9x) --- # These two carry the graded compressors -- _1_ through _9_ plus _99_ and _999_ # -- which is the widest API surface in the library and the best evidence that # the whole tree compiled rather than just the parts lzo1x needs. for n in 1 2 3 4 5 6 7 8 9 99 999; do decl "lzo/b-lzo1b_${n}_compress" "lzo1b_${n}_compress" "$HB" decl "lzo/c-lzo1c_${n}_compress" "lzo1c_${n}_compress" "$HCC" done for f in lzo1b_compress lzo1b_decompress lzo1b_decompress_safe; do decl "lzo/b-$f" "$f" "$HB" done for f in lzo1c_compress lzo1c_decompress lzo1c_decompress_safe; do decl "lzo/c-$f" "$f" "$HCC" done # -------------------------------------- family 4: lzo1, lzo1a, lzo1f, lzo2a --- # The remaining four. lzo1 and lzo1a are the original 1996 algorithms kept for # format compatibility; lzo1f and lzo2a are the compact-output variants. for f in lzo1_compress lzo1_decompress lzo1_99_compress; do decl "lzo/1-$f" "$f" "$H1" done for f in lzo1a_compress lzo1a_decompress lzo1a_99_compress; do decl "lzo/a-$f" "$f" "$HA" done for f in lzo1f_decompress lzo1f_decompress_safe lzo1f_1_compress lzo1f_999_compress; do decl "lzo/f-$f" "$f" "$HF" done for f in lzo2a_decompress lzo2a_decompress_safe lzo2a_999_compress; do decl "lzo/2a-$f" "$f" "$H2A" done # ---------------------------------------------------------------- pkg-config --- # The one place on target that states the port's version and its link line, and # what a future recipe's PKG_CHECK_MODULES reads. PC=$(cat "$pc" 2>/dev/null) has lzo/pc-name "Name: lzo2" "$PC" has lzo/pc-version "Version: 2.10" "$PC" has lzo/pc-desc "Description: LZO - a real-time data compression library" "$PC" has lzo/pc-libs "-llzo2" "$PC" # Shape fact 1 again, from the consumer's side: the include path a caller is # told to use ends in /lzo, matching where the headers actually are. has lzo/pc-cflags "Cflags: -I\${includedir}/lzo" "$PC" # libdir is compared EXACTLY, not as a substring: "libdir=/usr/lib" is also a # substring of the "/usr/lib/" a multiarch host writes, so the substring # form passes against a native library that is in the wrong place. (Measured on # gmp's suite during its native control run -- this is that lesson applied.) check_eq lzo/pc-libdir "/usr/lib" "$(awk -F= '/^libdir=/{print $2}' "$pc")" check_eq lzo/pc-prefix "/usr" "$(awk -F= '/^prefix=/{print $2}' "$pc")" check_eq lzo/pc-includedir "/usr/include" "$(awk -F= '/^includedir=/{print $2}' "$pc")" # ------------------------------------------------------------------ archive --- check lzo/lib-nonempty test -s "$lib" check_out lzo/lib-archive "!" head -c 8 "$lib" # A stub archive carrying the right index would still be orders of magnitude # smaller than the ~1 MB of compiled algorithm families this port produces. check lzo/lib-substantial sh -c '[ "$(wc -c < /usr/lib/liblzo2.a)" -gt 500000 ]' # Flatten only the symbol index -- see shape fact 4. idx=${TMPDIR:-/tmp}/bst-lzo.$$ head -c 16384 "$lib" | tr -c '\40-\176\n' ' ' > "$idx" 2>/dev/null # One compiled entry point per family plus the shared core, matched as whole # words. A partial compile that produced lzo1x but not lzo1b fails here, which # no header assertion above could catch -- the headers ship whether or not the # corresponding object compiled. for s in lzo1x_1_compress lzo1x_decompress lzo1x_decompress_safe \ lzo1x_999_compress lzo1x_optimize \ lzo1y_1_compress lzo1y_999_compress lzo1z_999_compress \ lzo1b_1_compress lzo1b_999_compress lzo1b_decompress_safe \ lzo1c_1_compress lzo1c_999_compress \ lzo1f_1_compress lzo1f_999_compress \ lzo1_compress lzo1_99_compress lzo1a_compress lzo2a_999_compress \ __lzo_init_v2 lzo_version lzo_version_string lzo_version_date \ lzo_adler32 lzo_crc32 lzo_get_crc32_table \ lzo_memcmp lzo_memcpy lzo_memmove lzo_memset \ _lzo_config_check __lzo_align_gap; do check "lzo/sym-$s" grep -qw "$s" "$idx" done rm -f "$idx" # ------------------------------------------------------- the arm witness ------ # Shape fact 3. Walk the ar structure to the first COMPILED member and read its # magic: 8-byte "!\n", then a 60-byte member header whose bytes 48..57 # carry the member size as decimal ASCII. Member 1 is always the symbol index # "/", so the first real member's header follows it (2-byte aligned) and its # data begins 60 bytes further on. Done by arithmetic rather than by scanning # the whole 1 MB file, and with no `ar` on this image (no binutils) there is no # shortcut. ARHDR=$(head -c 68 "$lib" 2>/dev/null | tail -c 60) IDXSZ=$(printf '%s' "$ARHDR" | cut -c49-58 | tr -d ' ') if [ -n "$IDXSZ" ] && [ "$IDXSZ" -gt 0 ] 2>/dev/null; then bst_ok lzo/ar-index-sized OFF=$((68 + IDXSZ)) [ $((OFF % 2)) -eq 1 ] && OFF=$((OFF + 1)) # The member name field is the first 16 bytes of that header; asserting it # confirms the arithmetic landed on a member header and not in the middle of # the index, so a wrong magic below cannot be blamed on a bad offset. MNAME=$(od -An -c -j "$OFF" -N 6 "$lib" 2>/dev/null | tr -d ' \n') has lzo/ar-first-member "lzo" "$MNAME" MAGIC=$(od -An -tx1 -j $((OFF + 60)) -N 4 "$lib" 2>/dev/null | tr -d ' \n') check_eq lzo/ar-member-bitcode 4243c0de "$MAGIC" # And it is NOT a native object: ELF would be 7f454c46, a finished wasm module # 0061736d. Asserting the two negatives makes the positive above a statement # about the arm rather than a magic number nobody can interpret. if [ "$MAGIC" = "7f454c46" ]; then bst_notok lzo/ar-not-elf "members are ELF -- this is a NATIVE build, not wasm32" else bst_ok lzo/ar-not-elf; fi if [ "$MAGIC" = "0061736d" ]; then bst_notok lzo/ar-not-wasm-module "members are finished wasm modules, not bitcode" else bst_ok lzo/ar-not-wasm-module; fi else bst_notok lzo/ar-index-sized "could not read ar symbol-index size from '$ARHDR'" fi # ------------------------------------------------------------------- policy --- # Static-only is distro policy and LZO's autotools build is configured with # --enable-shared, so a stray liblzo2.so here would mean the distro's # --disable-shared override was lost on this recipe. The main `lzo` package is # EMPTY for exactly this reason (its FILES is ${libdir}/lib*.so.*), which is why # only lzo-dev and lzo-staticdev are installed into the image. # The library's SHAPE rather than its absence -- BRINTOS-893's class. Asserting # that no shared object shipped encodes the distro-wide static premise BRINTOS-1740 # retires one recipe at a time, and this package is already allow-listed, so the # assertion fails it the moment the flip converts it correctly. What survives # conversion is that whatever shipped is usable. Swept under BRINTOS-1930. bst_lib_shape lzo/no-shared "/usr/lib/liblzo2.so" "/usr/lib/liblzo2.so.*" check lzo/no-libtool sh -c 'set -- /usr/lib/liblzo2.la; [ ! -e "$1" ]' check lzo/no-minilzo sh -c '[ ! -e /usr/lib/libminilzo.a ]' # lzop(1) is a SEPARATE poky recipe. This file used to assert its ABSENCE, which # was true when the lzo port landed and is not true now: lzop is on the image's # IMAGE_INSTALL and /usr/bin/lzop is present, so the assertion reported a correct # image as broken. Its subject was never lzo's to hold either -- what lzop ships # is 57-lzop.test's, and nothing about it is evidence for or against this port. # ------------------------------------------------ functional: the consumer ---- # Shape fact: with no CLI of its own and no on-target C compiler, the only way to # run THIS recipe's compiled code is through something that linked it. mkfs.jffs2 is # that something (see 96-mtd-utils-jffs2, which covers the tool's own side). # What is asserted HERE is the LIBRARY's contract -- that lzo compresses, and # that what it produces decompresses back to the original. if command -v mkfs.jffs2 >/dev/null 2>&1; then T=${TMPDIR:-/tmp}/bst-lzo-fn.$$ SRC=$T/src rm -rf "$T"; mkdir -p "$SRC" # ~66 KiB of highly compressible text from ten string doublings. A counting # loop of the same size costs thousands of builtin iterations on this runtime; # doubling costs ten. (96-mtd-utils-jffs2's recorded lesson, reapplied.) L='the quick brown fox jumps over the lazy dog 0123456789 ABCDEFGHIJ ' L="$L$L"; L="$L$L"; L="$L$L"; L="$L$L"; L="$L$L" L="$L$L"; L="$L$L"; L="$L$L"; L="$L$L"; L="$L$L" printf '%s' "$L" > "$SRC/fill.txt" # -X lzo ENABLES lzo (upstream registers it .disabled = 1, so it is compiled # in but OFF unless selected) and -x turns the other two off, so this image # can only have been produced by liblzo2. -t makes mkfs.jffs2 decompress every # block it wrote and compare it against the original, so a single invocation # is a full compress -> decompress -> verify round trip through our library. check lzo/fn-roundtrip \ mkfs.jffs2 -r "$SRC" -e 0x10000 -q -f -X lzo -x zlib -x rtime -t -o "$T/lzo.jffs2" # The round trip above proves correctness; this proves lzo actually COMPRESSED # rather than falling through to storing plain, which is the failure mode a # silently-unlinked compressor produces. check lzo/fn-store-none \ mkfs.jffs2 -r "$SRC" -e 0x10000 -q -f -m none -o "$T/none.jffs2" LZOSZ=$(size "$T/lzo.jffs2"); NONESZ=$(size "$T/none.jffs2") if [ -n "$LZOSZ" ] && [ -n "$NONESZ" ] && [ "$LZOSZ" -gt 512 ] \ && [ "$LZOSZ" -lt $((NONESZ / 4)) ]; then bst_ok lzo/fn-compressed else bst_notok lzo/fn-compressed "lzo image ${LZOSZ:-absent} vs uncompressed ${NONESZ:-absent}" fi rm -rf "$T" else bst_skip lzo/fn-roundtrip "mkfs.jffs2 not installed -- no in-image consumer links liblzo2" fi