Recovering data from crashed recording

Issues
1

Hello,
Thank you for the nice tool! Joulescope is really invaluable for our work.

I have got some jls files from recordings that got crashed (because I was using FAT-32 and the file got truncated at 4GB, my bad). Is it possible to recover measurements from there?
If I understand correctly, the raw measurements got recorded, but the file is in “streaming” format, while data_recorder.DataReader expects some navigational info. I’m now trying to put together a script that adds it post-hoc, but do you by any chance have it implemented already? If not, I’m going to share it when I have something working.

2

Hi @plotkin and welcome to the Joulescope forum! The JLS v1 code does have some recovery methods for file truncation. See here. Recovery is still a work in progress for JLS v2. See the “reliability” section in the readme. If you are using the Joulescope UI to record, then you are still using JLS v1.

Can you share the error you see when trying to open the truncated JLS file? If you are using the Joulescope UI, you should fine the error in the log. Try to open the JLS file. Then Help → View logs…. Close the UI and open the most recent log file.

Alternatively, you can try opening the file using python from the command line. You can using the included recording command:

pip3 install joulescope
joulescope recording {your_file.jls}
3

When trying to read file, I get:

WARNING:2022-06-27 22:21:56,012:datafile.py:517:joulescope.datafile:length was not written, use file length 4293777544
WARNING:2022-06-27 22:22:06,015:datafile.py:549:joulescope.datafile:invalid tag crc: 0x6ba650ae != 0x19ca2a98
WARNING:2022-06-27 22:22:11,636:datafile.py:549:joulescope.datafile:invalid tag crc: 0x6bb372ea != 0x28b51a32
Traceback (most recent call last):
  File "…/bin/joulescope", line 8, in <module>
    sys.exit(run())
  File "…/joulescope/entry_points/runner.py", line 105, in run
    return args.func(args)
  File "…/joulescope/entry_points/recording.py", line 79, in on_cmd
    r = DataReader().open(args.filename)
  File "…/joulescope/data_recorder.py", line 466, in open
    tag, value = self._f.peek()
  File "…/joulescope/datafile.py", line 592, in peek
    tag, flags, value, _ = self._read_tag()
  File "…/joulescope/datafile.py", line 544, in _read_tag
    raise ValueError('File too short')
ValueError: File too short

Looks like it’s trying to read the last (truncated) TLV block and fails.

I also tried patching the code to avoid the error:

$ diff datafie.py.orig datafie.py.patched 
544c544,545
<             raise ValueError('File too short')
---
>             return None, 0, b'', b''
>             #raise ValueError('File too short')
$ diff data_recorder.py.orig data_recorder.py.patched
611c611,612
<                 self._f.advance()
---
>                 try: self._f.advance()
>                 except OSError: break

But than the reader would only parse one second (a single block).

Also, here is a run of DataFileReader.pretty_print (i modified it to also print physical file offset):

length was not written, use file length 4293777544
20 SUB 1128
498 MJS 325
5f0 CLS id=0, type=0, start=0, end=0
  608 CLS id=1, type=0, start=0, end=8001688
    648 ABN 1600000
    187058 ABN 1600000
    30da68 ABN 1600000
    494478 ABN 1600000
    61ae88 ABN 1600000
  7a1898 CLE 10616
…
  ff739c30 CLS id=1, type=0, start=0, end=4293766848
    ff739c70 ABN 1600000
    ff8c0680 ABN 1600000
    ffa47090 ABN 1600000
    ffbcdaa0 ABN 1600000
    ffd544b0 ABN 1600000
  ffedaec0 CLE 10616
  ffedd848 CLS id=1, type=0, start=0, end=0

So one thing is that file is truncated, but is it also that all the blocks have start=0?

4

I just tested a good file, and pretty_print always display “start=0” for the collections. It’s been a long time since I looked at JLS v1, so not sure why off-hand. However, it’s not the problem. Here’s a good file:

SUB 1128
MJS 325
CLS id=0, type=0, start=0, end=54912176
  CLS id=1, type=0, start=0, end=8001688
    ABN 1600000
    ABN 1600000
    ABN 1600000
    ABN 1600000
    ABN 1600000
  CLE 10616
  CLS id=1, type=0, start=0, end=16012464
    ABN 1600000
    ABN 1600000
    ABN 1600000
    ABN 1600000
    ABN 1600000
  CLE 10616
  ...
  CLS id=1, type=0, start=0, end=54901544
    ABN 1600000
    ABN 1600000
    ABN 1600000
    ABN 1600000
    ABN 435200
  CLE 10616
CLE 0
MJS 36

I’m taking a look at the code now…

5

Ok, I think I figured out a soultion. The issue is that DataFileReader.skip() (which is used by DataReader._sample_count() recovery routine) expects correct offsets in collection TLVs, which are not here, so it tries to skip over first collection to some random garbage. The fix is to use advance() for collections as well, it will be slower, but working.
Another issue is trying to read truncated reductions from the last collection. Quick fix is just to omit those (truncating file to the last CLE also works, but is a bit more hussle).
Here is a patch that fixes both:

$ diff data_recorder.py.orig data_recorder.py.patched
608d607
<                 self._f.skip()
610,611c609
<             else:
<                 self._f.advance()
---
>             self._f.advance()
860c858,859
<                 raise ValueError('invalid file format: not collection end')
---
>                 return None
>                 # raise ValueError('invalid file format: not collection end')

$ diff datafie.py.orig datafie.py.patched
544c544,545
<             raise ValueError('File too short')
---
>             return None, 0, b'', b''
>             #raise ValueError('File too short')

Reduction data might be off, but at least we have data raw.
Thank you for you help, @mliberty and hope it helps someone.
[EDIT] duplicated datafile.py patch in the solution post to avoid confusion.

1 Like
6

Nice! I agree that your patch should get the job done. If you want a completely fixed file, you can use jls_recode to read the truncated file with your patch and then write out a fixed file.