Splitting jls v2 file into multiple

Hi @mliberty
I do this as follow up from Automation of plotting long-term records - #18 by mliberty in a new ticket.
It seems your suggestions are covering jls 1.
I tried to adapt it to jls 2. Here my snipped:

Hi
I looked into the code and it looks like it handles jls version 1 only.
I tried:

r = Reader(args.input)
signals = [s for s in r.signals.values() if s.name == ‘current’]
if not len(signals):
log.error(’“current” signal not found’)
return 1
s = signals[0]
i_old = 0
i= 0
sample_count = s.length
block_size = int(s.sample_rate)
print(f’samples={sample_count}, fs={s.sample_rate}’)
block_count = (sample_count + block_size - 1) // block_size
for block in range(block_count):
if (block_count // 1800)==0 or block == 0:
filename = “outfile_jls2_{}.jls”.format(block)
writer = DataRecorder(filename, calibration=None, user_data=r.user_data)
i += 1
offset = block * block_size
offset_next = offset + block_size
if offset_next > sample_count:
offset_next = sample_count
data = range.samples_get(offset, offset_next, ‘samples’)
writer.insert(data)
if i != i_old:
writer.close()
#progress(block, block_count - 1)
r.close()

I get the error

Traceback (most recent call last):
File “C:\Users\lukas.kempf\AppData\Local\JetBrains\Toolbox\apps\PyCharm-C\ch-0\213.6777.50\plugins\python-ce\helpers\pydev\pydevd.py”, line 1483, in exec
pydev_imports.execfile(file, globals, locals) # execute the script
File “C:\Users\lukas.kempf\AppData\Local\JetBrains\Toolbox\apps\PyCharm-C\ch-0\213.6777.50\plugins\python-ce\helpers\pydev_pydev_imps_pydev_execfile.py”, line 18, in execfile
exec(compile(contents+"\n", file, ‘exec’), glob, loc)
File "C:/code/pyjoulescope_examples
/bin/jls_plot.py", line 834, in
sys.exit(run())
File “C:/code/pyjoulescope_examples_/bin/jls_plot.py”, line 369, in run
writer = DataRecorder(filename, calibration=None, user_data=r.user_data)
File “C:\tools\miniconda3\envs\pyjsui\lib\site-packages\joulescope\data_recorder.py”, line 117, in init
b = json.dumps(user_data).encode(‘utf-8’)
File “C:\tools\miniconda3\envs\pyjsui\lib\json_init_.py”, line 231, in dumps
return _default_encoder.encode(obj)
File “C:\tools\miniconda3\envs\pyjsui\lib\json\encoder.py”, line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File “C:\tools\miniconda3\envs\pyjsui\lib\json\encoder.py”, line 257, in iterencode
return _iterencode(o, 0)
File “C:\tools\miniconda3\envs\pyjsui\lib\json\encoder.py”, line 179, in default
raise TypeError(f’Object of type {o.class.name} ’
TypeError: Object of type builtin_function_or_method is not JSON serializable
Do you have a hint?

Hi @lukGWK

The API for JLS v2 Reader is very different from JLS v1. Also, DataRecorder is JLS v1, not JLS v2.

I am not clear on what you are trying to do. Are you trying to process a JLS v2 file in blocks? Are you trying to extract a snippet of a large JLS v2 file and save it to a smaller JLS v2 file?

With JLS v2 Reader, you probably want to identify the signal(s) of interest, which also returns everything you need to know about the signal. See pyjls.structs.SignalDef which shows the fields you will expect from Reader.signals. You can then call Reader.fsr to get blocks of the signals of interest to process.

I try to record 2MSps for 1 day and split the file which is x*10GB to 1GB files to process it. I had an error when opening the file which was bigger than the amount of RAM available.

Going to the original question, I think that you can fix the crash by not providing user_data to DataRecorder. Reader.user_data is actually a method, so it’s not JSON serializable.

However, I am unsure what the code snippet above has to do with processing sections of data with JLS v2. For example, range is not even defined in the snippet above, so range.samples_get will definitely fail. With JLS v2 Reader, samples_get is not a method.

The JLS v2 Reader does not have a lot of examples yet. You definitely need to be looking at the source code, but it’s pretty simple. You need to use Reader.fsr to get blocks of data. You have to call Reader.fsr once for each signal of interest in each block.

Does this make sense?