[PATCH 0/2] fs/squashfs: Add new decompression algorithms

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[PATCH 0/2] fs/squashfs: Add new decompression algorithms

Joao Marcos Costa
Hello,

Following the SquashFS support, this series adds support for LZO and ZSTD
algorithms. The only compression type enabled by default is ZLIB, so LZO
and ZSTD need to be manually selected.

Joao Marcos Costa (2):
  fs/squashfs: add support for LZO decompression
  fs/squashfs: add support for ZSTD decompression

 fs/squashfs/sqfs_decompressor.c | 60 +++++++++++++++++++++++++++++++--
 1 file changed, 58 insertions(+), 2 deletions(-)

--
2.17.1

Reply | Threaded
Open this post in threaded view
|

[PATCH 1/2] fs/squashfs: add support for LZO decompression

Joao Marcos Costa
Add call to lzo's lzo1x_decompress_safe() and rename source length
parameter from 'lenp' to 'src_len'.

Signed-off-by: Joao Marcos Costa <[hidden email]>
---
 fs/squashfs/sqfs_decompressor.c | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/fs/squashfs/sqfs_decompressor.c b/fs/squashfs/sqfs_decompressor.c
index 09ca6cf6d0..9285df5d3b 100644
--- a/fs/squashfs/sqfs_decompressor.c
+++ b/fs/squashfs/sqfs_decompressor.c
@@ -9,6 +9,11 @@
 #include <stdint.h>
 #include <stdio.h>
 #include <stdlib.h>
+
+#if IS_ENABLED(CONFIG_LZO)
+#include <linux/lzo.h>
+#endif
+
 #if IS_ENABLED(CONFIG_ZLIB)
 #include <u-boot/zlib.h>
 #endif
@@ -35,20 +40,32 @@ static void zlib_decompression_status(int ret)
 #endif
 
 int sqfs_decompress(u16 comp_type, void *dest, unsigned long *dest_len,
-    void *source, u32 lenp)
+    void *source, u32 src_len)
 {
  int ret = 0;
 
  switch (comp_type) {
 #if IS_ENABLED(CONFIG_ZLIB)
  case SQFS_COMP_ZLIB:
- ret = uncompress(dest, dest_len, source, lenp);
+ ret = uncompress(dest, dest_len, source, src_len);
  if (ret) {
  zlib_decompression_status(ret);
  return -EINVAL;
  }
 
  break;
+#endif
+#if IS_ENABLED(CONFIG_LZO)
+ case SQFS_COMP_LZO: {
+ size_t lzo_dest_len = *dest_len;
+ ret = lzo1x_decompress_safe(source, src_len, dest, &lzo_dest_len);
+ if (ret) {
+ printf("LZO decompression failed. Error code: %d\n", ret);
+ return -EINVAL;
+ }
+
+ break;
+ }
 #endif
  default:
  printf("Error: unknown compression type.\n");
--
2.17.1

Reply | Threaded
Open this post in threaded view
|

[PATCH 2/2] fs/squashfs: add support for ZSTD decompression

Joao Marcos Costa
In reply to this post by Joao Marcos Costa
Add call to ZSTD's ZSTD_decompressDCtx(). In this use case, the caller
can upper bound the decompressed size, which will be the SquashFS data
block (or metadata block) size, so there is no need to use streaming
API.

Signed-off-by: Joao Marcos Costa <[hidden email]>
---
 fs/squashfs/sqfs_decompressor.c | 39 +++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/fs/squashfs/sqfs_decompressor.c b/fs/squashfs/sqfs_decompressor.c
index 9285df5d3b..b5a9d92808 100644
--- a/fs/squashfs/sqfs_decompressor.c
+++ b/fs/squashfs/sqfs_decompressor.c
@@ -18,6 +18,10 @@
 #include <u-boot/zlib.h>
 #endif
 
+#if IS_ENABLED(CONFIG_ZSTD)
+#include <linux/zstd.h>
+#endif
+
 #include "sqfs_decompressor.h"
 #include "sqfs_filesystem.h"
 #include "sqfs_utils.h"
@@ -39,6 +43,31 @@ static void zlib_decompression_status(int ret)
 }
 #endif
 
+#if IS_ENABLED(CONFIG_ZSTD)
+static int sqfs_zstd_decompress(void *dest, unsigned long dest_len,
+ void *source, u32 src_len)
+{
+ void *workspace;
+ ZSTD_DCtx *ctx;
+ size_t wsize;
+ int ret;
+
+ wsize = ZSTD_DCtxWorkspaceBound();
+
+ workspace = malloc(wsize);
+ if (!workspace)
+ return -ENOMEM;
+
+ ctx = ZSTD_initDCtx(workspace, wsize);
+
+ ret = ZSTD_decompressDCtx(ctx, dest, dest_len, source, src_len);
+
+ free(workspace);
+
+ return ZSTD_isError(ret);
+}
+#endif /* CONFIG_ZSTD */
+
 int sqfs_decompress(u16 comp_type, void *dest, unsigned long *dest_len,
     void *source, u32 src_len)
 {
@@ -67,6 +96,16 @@ int sqfs_decompress(u16 comp_type, void *dest, unsigned long *dest_len,
  break;
  }
 #endif
+#if IS_ENABLED(CONFIG_ZSTD)
+ case SQFS_COMP_ZSTD:
+ ret = sqfs_zstd_decompress(dest, *dest_len, source, src_len);
+ if (ret) {
+ printf("ZSTD Error code: %d\n", ZSTD_getErrorCode(ret));
+ return -EINVAL;
+ }
+
+ break;
+#endif /* CONFIG_ZSTD */
  default:
  printf("Error: unknown compression type.\n");
  return -EINVAL;
--
2.17.1

Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 2/2] fs/squashfs: add support for ZSTD decompression

Thomas Petazzoni
On Tue, 11 Aug 2020 15:17:55 +0200
Joao Marcos Costa <[hidden email]> wrote:

> +#if IS_ENABLED(CONFIG_ZSTD)
> +static int sqfs_zstd_decompress(void *dest, unsigned long dest_len,
> + void *source, u32 src_len)
> +{
> + void *workspace;
> + ZSTD_DCtx *ctx;
> + size_t wsize;
> + int ret;
> +
> + wsize = ZSTD_DCtxWorkspaceBound();

So apparently this "workspace" has a constant size, which does not
depend on the size of the input buffer. Correct ?

If that's the case, can we instead allocate it once for all in
sqfs_probe() if the filesystem is zstd-compressed ? I.e perhaps
sqfs_probe() could call a sqfs_decompressor_init() function,
implemented in sqfs_decompressor.c, which will do this sort of
initialization. And of course this memory area would be freed up in
sqfs_close(), calling a sqfs_decompressor_{deinit,cleanup,close}
function.

Best regards,

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 2/2] fs/squashfs: add support for ZSTD decompression

Joao Marcos Costa
On Tue, 11 Aug 2020 15:29:51 +0200
Thomas Petazzoni <[hidden email]> wrote:

> On Tue, 11 Aug 2020 15:17:55 +0200
> Joao Marcos Costa <[hidden email]> wrote:
>
> > +#if IS_ENABLED(CONFIG_ZSTD)
> > +static int sqfs_zstd_decompress(void *dest, unsigned long dest_len,
> > + void *source, u32 src_len)
> > +{
> > + void *workspace;
> > + ZSTD_DCtx *ctx;
> > + size_t wsize;
> > + int ret;
> > +
> > + wsize = ZSTD_DCtxWorkspaceBound();  
>
> So apparently this "workspace" has a constant size, which does not
> depend on the size of the input buffer. Correct ?

Yes, correct.

> If that's the case, can we instead allocate it once for all in
> sqfs_probe() if the filesystem is zstd-compressed ? I.e perhaps
> sqfs_probe() could call a sqfs_decompressor_init() function,
> implemented in sqfs_decompressor.c, which will do this sort of
> initialization. And of course this memory area would be freed up in
> sqfs_close(), calling a sqfs_decompressor_{deinit,cleanup,close}
> function.

Sure, it can be done. I will prepare a v2 addressing your suggestion.
Thanks!

> Best regards,
>
> Thomas

Best regards,
Joao
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/2] fs/squashfs: Add new decompression algorithms

Tom Rini-4
In reply to this post by Joao Marcos Costa
On Tue, Aug 11, 2020 at 03:17:53PM +0200, Joao Marcos Costa wrote:

> Hello,
>
> Following the SquashFS support, this series adds support for LZO and ZSTD
> algorithms. The only compression type enabled by default is ZLIB, so LZO
> and ZSTD need to be manually selected.
>
> Joao Marcos Costa (2):
>   fs/squashfs: add support for LZO decompression
>   fs/squashfs: add support for ZSTD decompression
>
>  fs/squashfs/sqfs_decompressor.c | 60 +++++++++++++++++++++++++++++++--
>  1 file changed, 58 insertions(+), 2 deletions(-)
Can we add tests for this to sandbox?  Thanks!

--
Tom

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/2] fs/squashfs: Add new decompression algorithms

Joao Marcos Costa
On Tue, 11 Aug 2020 11:13:41 -0400
Tom Rini <[hidden email]> wrote:

> On Tue, Aug 11, 2020 at 03:17:53PM +0200, Joao Marcos Costa wrote:
>
> > Hello,
> >
> > Following the SquashFS support, this series adds support for LZO
> > and ZSTD algorithms. The only compression type enabled by default
> > is ZLIB, so LZO and ZSTD need to be manually selected.
> >
> > Joao Marcos Costa (2):
> >   fs/squashfs: add support for LZO decompression
> >   fs/squashfs: add support for ZSTD decompression
> >
> >  fs/squashfs/sqfs_decompressor.c | 60
> > +++++++++++++++++++++++++++++++-- 1 file changed, 58 insertions(+),
> > 2 deletions(-)  
>
> Can we add tests for this to sandbox?  Thanks!

Sure, I will enable the lzo and zstd configs for, and I will make some
actual changes to the test scripts I added, because right now they are
not really "scalable" (thinking of new compression algorithms).

Best regards,
Joao
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/2] fs/squashfs: Add new decompression algorithms

Joao Marcos Costa
In reply to this post by Tom Rini-4
On Tue, 11 Aug 2020 11:13:41 -0400
Tom Rini <[hidden email]> wrote:

> On Tue, Aug 11, 2020 at 03:17:53PM +0200, Joao Marcos Costa wrote:
>
> > Hello,
> >
> > Following the SquashFS support, this series adds support for LZO
> > and ZSTD algorithms. The only compression type enabled by default
> > is ZLIB, so LZO and ZSTD need to be manually selected.
> >
> > Joao Marcos Costa (2):
> >   fs/squashfs: add support for LZO decompression
> >   fs/squashfs: add support for ZSTD decompression
> >
> >  fs/squashfs/sqfs_decompressor.c | 60
> > +++++++++++++++++++++++++++++++-- 1 file changed, 58 insertions(+),
> > 2 deletions(-)  
>
> Can we add tests for this to sandbox?  Thanks!
>

Hello,

Is there any way to check the package version with the python API? I
already use 'requiredtool' marker, but apparently it is not enough. My
host machine (Ubuntu 18.04) has mksquashfs at its version 4.3, without
zstd support. However, v4.4 has such support, and I had to download it
and build it myself to use zstd compression. The tests will fail if the
user has an older version of mksquashfs installed. Would you have any
suggestion on how to proceed?

Best regards,
Joao
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/2] fs/squashfs: Add new decompression algorithms

Tom Rini-4
On Wed, Aug 12, 2020 at 05:13:06PM +0200, Joao Marcos Costa wrote:

> On Tue, 11 Aug 2020 11:13:41 -0400
> Tom Rini <[hidden email]> wrote:
>
> > On Tue, Aug 11, 2020 at 03:17:53PM +0200, Joao Marcos Costa wrote:
> >
> > > Hello,
> > >
> > > Following the SquashFS support, this series adds support for LZO
> > > and ZSTD algorithms. The only compression type enabled by default
> > > is ZLIB, so LZO and ZSTD need to be manually selected.
> > >
> > > Joao Marcos Costa (2):
> > >   fs/squashfs: add support for LZO decompression
> > >   fs/squashfs: add support for ZSTD decompression
> > >
> > >  fs/squashfs/sqfs_decompressor.c | 60
> > > +++++++++++++++++++++++++++++++-- 1 file changed, 58 insertions(+),
> > > 2 deletions(-)  
> >
> > Can we add tests for this to sandbox?  Thanks!
> >
>
> Hello,
>
> Is there any way to check the package version with the python API? I
> already use 'requiredtool' marker, but apparently it is not enough. My
> host machine (Ubuntu 18.04) has mksquashfs at its version 4.3, without
> zstd support. However, v4.4 has such support, and I had to download it
> and build it myself to use zstd compression. The tests will fail if the
> user has an older version of mksquashfs installed. Would you have any
> suggestion on how to proceed?
This sounds like the cases where we have to build the tools we need.
There's an example in .travis.yml for QEMU already, and we use
https://gitlab.denx.de/u-boot/gitlab-ci-runner/ for Azure/GitLab.
Testing those with your own container does require setting up an Azure
account or using your own GitLab + runner.

That said, if you unit test things and are confident it won't blow up
when I test it, that can be good enough.  Thanks!

--
Tom

signature.asc (673 bytes) Download Attachment