Root/
1 | |
2 | XZ data compression in Linux |
3 | ============================ |
4 | |
5 | Introduction |
6 | |
7 | XZ is a general purpose data compression format with high compression |
8 | ratio and relatively fast decompression. The primary compression |
9 | algorithm (filter) is LZMA2. Additional filters can be used to improve |
10 | compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters |
11 | improve compression ratio of executable data. |
12 | |
13 | The XZ decompressor in Linux is called XZ Embedded. It supports |
14 | the LZMA2 filter and optionally also BCJ filters. CRC32 is supported |
15 | for integrity checking. The home page of XZ Embedded is at |
16 | <http://tukaani.org/xz/embedded.html>, where you can find the |
17 | latest version and also information about using the code outside |
18 | the Linux kernel. |
19 | |
20 | For userspace, XZ Utils provide a zlib-like compression library |
21 | and a gzip-like command line tool. XZ Utils can be downloaded from |
22 | <http://tukaani.org/xz/>. |
23 | |
24 | XZ related components in the kernel |
25 | |
26 | The xz_dec module provides XZ decompressor with single-call (buffer |
27 | to buffer) and multi-call (stateful) APIs. The usage of the xz_dec |
28 | module is documented in include/linux/xz.h. |
29 | |
30 | The xz_dec_test module is for testing xz_dec. xz_dec_test is not |
31 | useful unless you are hacking the XZ decompressor. xz_dec_test |
32 | allocates a char device major dynamically to which one can write |
33 | .xz files from userspace. The decompressed output is thrown away. |
34 | Keep an eye on dmesg to see diagnostics printed by xz_dec_test. |
35 | See the xz_dec_test source code for the details. |
36 | |
37 | For decompressing the kernel image, initramfs, and initrd, there |
38 | is a wrapper function in lib/decompress_unxz.c. Its API is the |
39 | same as in other decompress_*.c files, which is defined in |
40 | include/linux/decompress/generic.h. |
41 | |
42 | scripts/xz_wrap.sh is a wrapper for the xz command line tool found |
43 | from XZ Utils. The wrapper sets compression options to values suitable |
44 | for compressing the kernel image. |
45 | |
46 | For kernel makefiles, two commands are provided for use with |
47 | $(call if_needed). The kernel image should be compressed with |
48 | $(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2 |
49 | dictionary. It will also append a four-byte trailer containing the |
50 | uncompressed size of the file, which is needed by the boot code. |
51 | Other things should be compressed with $(call if_needed,xzmisc) |
52 | which will use no BCJ filter and 1 MiB LZMA2 dictionary. |
53 | |
54 | Notes on compression options |
55 | |
56 | Since the XZ Embedded supports only streams with no integrity check or |
57 | CRC32, make sure that you don't use some other integrity check type |
58 | when encoding files that are supposed to be decoded by the kernel. With |
59 | liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32 |
60 | when encoding. With the xz command line tool, use --check=none or |
61 | --check=crc32. |
62 | |
63 | Using CRC32 is strongly recommended unless there is some other layer |
64 | which will verify the integrity of the uncompressed data anyway. |
65 | Double checking the integrity would probably be waste of CPU cycles. |
66 | Note that the headers will always have a CRC32 which will be validated |
67 | by the decoder; you can only change the integrity check type (or |
68 | disable it) for the actual uncompressed data. |
69 | |
70 | In userspace, LZMA2 is typically used with dictionary sizes of several |
71 | megabytes. The decoder needs to have the dictionary in RAM, thus big |
72 | dictionaries cannot be used for files that are intended to be decoded |
73 | by the kernel. 1 MiB is probably the maximum reasonable dictionary |
74 | size for in-kernel use (maybe more is OK for initramfs). The presets |
75 | in XZ Utils may not be optimal when creating files for the kernel, |
76 | so don't hesitate to use custom settings. Example: |
77 | |
78 | xz --check=crc32 --lzma2=dict=512KiB inputfile |
79 | |
80 | An exception to above dictionary size limitation is when the decoder |
81 | is used in single-call mode. Decompressing the kernel itself is an |
82 | example of this situation. In single-call mode, the memory usage |
83 | doesn't depend on the dictionary size, and it is perfectly fine to |
84 | use a big dictionary: for maximum compression, the dictionary should |
85 | be at least as big as the uncompressed data itself. |
86 | |
87 | Future plans |
88 | |
89 | Creating a limited XZ encoder may be considered if people think it is |
90 | useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at |
91 | the fastest settings, so it isn't clear if LZMA2 encoder is wanted |
92 | into the kernel. |
93 | |
94 | Support for limited random-access reading is planned for the |
95 | decompression code. I don't know if it could have any use in the |
96 | kernel, but I know that it would be useful in some embedded projects |
97 | outside the Linux kernel. |
98 | |
99 | Conformance to the .xz file format specification |
100 | |
101 | There are a couple of corner cases where things have been simplified |
102 | at expense of detecting errors as early as possible. These should not |
103 | matter in practice all, since they don't cause security issues. But |
104 | it is good to know this if testing the code e.g. with the test files |
105 | from XZ Utils. |
106 | |
107 | Reporting bugs |
108 | |
109 | Before reporting a bug, please check that it's not fixed already |
110 | at upstream. See <http://tukaani.org/xz/embedded.html> to get the |
111 | latest code. |
112 | |
113 | Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on |
114 | Freenode and talk to Larhzu. I don't actively read LKML or other |
115 | kernel-related mailing lists, so if there's something I should know, |
116 | you should email to me personally or use IRC. |
117 | |
118 | Don't bother Igor Pavlov with questions about the XZ implementation |
119 | in the kernel or about XZ Utils. While these two implementations |
120 | include essential code that is directly based on Igor Pavlov's code, |
121 | these implementations aren't maintained nor supported by him. |
122 |
Branches:
ben-wpan
ben-wpan-stefan
javiroman/ks7010
jz-2.6.34
jz-2.6.34-rc5
jz-2.6.34-rc6
jz-2.6.34-rc7
jz-2.6.35
jz-2.6.36
jz-2.6.37
jz-2.6.38
jz-2.6.39
jz-3.0
jz-3.1
jz-3.11
jz-3.12
jz-3.13
jz-3.15
jz-3.16
jz-3.18-dt
jz-3.2
jz-3.3
jz-3.4
jz-3.5
jz-3.6
jz-3.6-rc2-pwm
jz-3.9
jz-3.9-clk
jz-3.9-rc8
jz47xx
jz47xx-2.6.38
master
Tags:
od-2011-09-04
od-2011-09-18
v2.6.34-rc5
v2.6.34-rc6
v2.6.34-rc7
v3.9