Hi
I looked into the code and it looks like it handles jls version 1 only.
I tried:
r = Reader(args.input)
signals = [s for s in r.signals.values() if s.name == ‘current’]
if not len(signals):
log.error(‘“current” signal not found’)
return 1
s = signals[0]
i_old = 0
i= 0
sample_count = s.length
block_size = int(s.sample_rate)
print(f’samples={sample_count}, fs={s.sample_rate}')
block_count = (sample_count + block_size - 1) // block_size
for block in range(block_count):
if (block_count // 1800)==0 or block == 0:
filename = “outfile_jls2_{}.jls”.format(block)
writer = DataRecorder(filename, calibration=None, user_data=r.user_data)
i += 1
offset = block * block_size
offset_next = offset + block_size
if offset_next > sample_count:
offset_next = sample_count
data = range.samples_get(offset, offset_next, ‘samples’)
writer.insert(data)
if i != i_old:
writer.close() #progress(block, block_count - 1)
r.close()
I get the error
Traceback (most recent call last):
File “C:\Users\lukas.kempf\AppData\Local\JetBrains\Toolbox\apps\PyCharm-C\ch-0\213.6777.50\plugins\python-ce\helpers\pydev\pydevd.py”, line 1483, in exec
pydev_imports.execfile(file, globals, locals) # execute the script
File “C:\Users\lukas.kempf\AppData\Local\JetBrains\Toolbox\apps\PyCharm-C\ch-0\213.6777.50\plugins\python-ce\helpers\pydev_pydev_imps_pydev_execfile.py”, line 18, in execfile
exec(compile(contents+“\n”, file, ‘exec’), glob, loc)
File "C:/code/pyjoulescope_examples/bin/jls_plot.py", line 834, in
sys.exit(run())
File “C:/code/pyjoulescope_examples_/bin/jls_plot.py”, line 369, in run
writer = DataRecorder(filename, calibration=None, user_data=r.user_data)
File “C:\tools\miniconda3\envs\pyjsui\lib\site-packages\joulescope\data_recorder.py”, line 117, in init
b = json.dumps(user_data).encode(‘utf-8’)
File “C:\tools\miniconda3\envs\pyjsui\lib\json_init_.py”, line 231, in dumps
return _default_encoder.encode(obj)
File “C:\tools\miniconda3\envs\pyjsui\lib\json\encoder.py”, line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File “C:\tools\miniconda3\envs\pyjsui\lib\json\encoder.py”, line 257, in iterencode
return _iterencode(o, 0)
File “C:\tools\miniconda3\envs\pyjsui\lib\json\encoder.py”, line 179, in default
raise TypeError(f’Object of type {o.class.name} ’
TypeError: Object of type builtin_function_or_method is not JSON serializable
Do you have a hint?
The API for JLS v2 Reader is very different from JLS v1. Also, DataRecorder is JLS v1, not JLS v2.
I am not clear on what you are trying to do. Are you trying to process a JLS v2 file in blocks? Are you trying to extract a snippet of a large JLS v2 file and save it to a smaller JLS v2 file?
With JLS v2 Reader, you probably want to identify the signal(s) of interest, which also returns everything you need to know about the signal. See pyjls.structs.SignalDef which shows the fields you will expect from Reader.signals. You can then call Reader.fsr to get blocks of the signals of interest to process.
I try to record 2MSps for 1 day and split the file which is x*10GB to 1GB files to process it. I had an error when opening the file which was bigger than the amount of RAM available.
Going to the original question, I think that you can fix the crash by not providing user_data to DataRecorder. Reader.user_data is actually a method, so it’s not JSON serializable.
However, I am unsure what the code snippet above has to do with processing sections of data with JLS v2. For example, range is not even defined in the snippet above, so range.samples_get will definitely fail. With JLS v2 Reader, samples_get is not a method.
The JLS v2 Reader does not have a lot of examples yet. You definitely need to be looking at the source code, but it’s pretty simple. You need to use Reader.fsr to get blocks of data. You have to call Reader.fsr once for each signal of interest in each block.
Hi @lukGWF - You can easily do this in the Joulescope UI using dual markers. Position the dual markers. Right-click on one of the dual markers, then select Export.
Are you still looking to write a python script for this? JLS files support unaligned data. The first step is to determine the start and end time of the region in UTC time. Do you have a way to do this? Do you need to also analyze the JLS file to determine the regions to extract?
I know I could do it in the UI. But I would appreciate to have a scriptable solution.
I have this method:
def extract_snippet(input_file_path, output_file_path, start_time, end_time):
"""
Extracts a snippet from a JLS file and saves it to a new file.
:param input_file_path: Path to the input JLS file
:param output_file_path: Path to save the output snippet
:param start_time: Start time of the snippet in seconds
:param end_time: End time of the snippet in seconds
"""
# Open the .jls file and extract the snippet
with Reader(input_file_path) as r:
signal = r.signal_lookup('current')
start_idx = start_time * signal.sample_rate
end_idx = end_time * signal.sample_rate
data = r.fsr(signal.signal_id, start_idx, end_idx)
fre = plt.figure()
ax_i_fre = fre.add_subplot(1, 1, 1)
ax_i_fre.grid(True)
incr = 100
y = data[: , 0]
x = np.arange(0, len(y), dtype=np.float64) * (incr / signal.sample_rate)
ax_i_fre.plot(x, data)
plt.show()
I couldn’t figure out how:
to put it back into a jls v2? I was looking in the pyjoulescope_ui exporter.py _run()?
to reference using UTC.
Do you mean UTC when it was recorded as absolute time?
Hi @mliberty,
2 infos found while using as a python module:
extract.py:
line 114 is: t64_start = t64_start_limit + int(time64.duration_to_seconds(args.start) * time64.SECOND)
changed to: t64_start = time64.duration_to_seconds(args.start)
line 125 is: t64_stop_limit_str = time64.as_datetime(t64_start_limit).isoformat()
changed to: t64_stop_limit_str = time64.as_datetime(t64_stop_limit).isoformat()
the call extract.py:cmd()
needs the argument prep:
end_time = stats['stop_time']['value']
start_time = stats['start_time']['value']
start_time = end_time - last_n_secs
start_time_iso = datetime.fromtimestamp(start_time).isoformat()
end_time_iso = datetime.fromtimestamp(end_time).isoformat()
if last_n_secs > end_time:
length_s = end_time
if start_time > end_time:
print('End time should be >= start time')
sys.exit(-1)
parser = argparse.ArgumentParser(description='Extract data from JLS file')
parser_config(parser)
args = parser.parse_args([
'--start', str(start_time_iso), <<< iso time is expected here
'--stop', str(end_time_iso), <<< iso time is expected here
input_file_path,
output_file_path
])
I agree with the change on line 125. Fortunately, t64_stop_limit_str is only used for error messages, so behavior is unchanged.
I think that line 114 is correct as originally given. --start and --stop are either ISO time strings or offsets from the start. Line 114 converts from offset in seconds to a time64 value. The t64_start_limit is used to set the start time at the first point that all signals in the JLS file contain data. The * time64.SECOND converts from seconds to time64. What about this did you find incorrect?
I actually think that line 123 is incorrect . It should match 114, like this:
...
from pyjls.entry_points.extract import on_cmd as slice_jls
...
which instantiate:
r = Reader()
...
w = Writer()
leads to the error:
WARNING pyjls.c:raw.c:146 file header length 0, not closed gracefully
WARNING pyjls.c:reader.c:100 not properly closed
WARNING pyjls.c:backend_win.c:72 open failed with 13: filename=20240709_163908_MITTO_eu.jls, mode=a
analyze the sliced jls
code of the loadJLSFile() method:
log.info(f"analyze.py: loadJLSFile(jls_file_path={jls_file_path}, incr={incr}, plot_point_count={plot_point_count})")
try:
prefix = None
if not os.path.isfile(jls_file_path):
raise FileNotFoundError()
else:
with open(jls_file_path, 'rb') as f:
prefix = f.read(16)
if prefix == _V2_PREFIX:
log.info(f'Reading JLS v2: {jls_file_path}')
r = Reader(jls_file_path)
signals = [s for s in r.signals.values() if s.name == 'current']
for s in r.sources.values():
if s.source_id == 1:
rec_model = s.model
rec_sn = s.serial_number
if not len(signals):
log.error('"current" signal not found')
return 1
s = signals[0]
leads to this error:
ERROR agent_cpm.analyze:analyze.py:360 open IO[4]: Input/output error
ERROR agent_cpm.analyze:analyze.py:361 Traceback (most recent call last):
File "C:\Users\nitdrbill\AppData\Local\pypoetry\Cache\virtualenvs\bromine-XwSBaFod-py3.9\lib\site-packages\agent_cpm\analyze.py", line 330, in loadJLSFile
r = Reader(jls_file_path)
File "pyjls/binding.pyx", line 549, in pyjls.binding.Reader.__init__
File "pyjls/binding.pyx", line 269, in pyjls.binding._handle_rc
RuntimeError: open IO[4]: Input/output error
I assume during the slice process a Reader and a Writer is called with c-bindings. This is not being closed after executing the method.
How could I force it?
Is there something else I missed?
Hi @lukGWF - If I understand correctly, it is the first Reader() in your slice_jls file import that gives the warnings about not being closed correctly. Is that right?
If so, are you sure that the pyjoulescope_driver capture command completes before you start slice_jls?
What exactly is pyjoulescope_driver capture? Do you mean the pyjoulescope_driverrecord entry point?
. It should match 114, like this: