Add "notes" to jls file at/after capture

lukGWF · July 16, 2024, 6:47pm

Hi,
I would like to add the voltage level before/after recording. Viewing them I can already python -m pyjls info [filename.jls] (related. But adding them either when using python -m pyjoulescope_driver record ... or before/after I was not able to do. When I add the information in the UI at start of recording I find it using the mentioned variant. In the corresponding .anno.jls is neither length nor “notes” included. What is the purpose of this file?

mliberty · July 16, 2024, 7:15pm

Hi @lukGWF,

So, why not just record voltage along with current? This eliminates the need to add custom annotations.

I think that there are a few questions in here regarding JLS files and notes:

Can I add “notes” after a capture using the Joulescope UI?
No, not in any existing version of the Joulescope UI (now 1.1.10). You can add notes before in the UI, but not after.
Using pyjls, can I add notes whenever I want?
Yes. You can actually add any text, json, or binary blob you want with Writer.user_data(). The UI uses chunk_meta 0x0000 for notes and 0x8001 for the UI metadata. You can use any other value to write whatever you want. However, these will be ignored by the UI. You can also add annotations using Writer.annotation(). See the pyjls Writer docs.
Using pyjoulescope_driver.record.Record, can I add notes whenever I want?
With the existing code as of 2024-07-16, it’s a hack, but you can call Record._wr.user_data(). You may be better off writing your own record procedure / class to make this cleaner using (2).
What is the purpose of the <base>.anno.jls file?
This file stores the annotations (vertical markers, horizontal markers, text annotations) that you add in the Joulescope UI. The UI saves these separately so that saving annotations does not require modifying the captured data. I am a huge proponent of never modifying captured data. The UI loads both the base <base>.jls file and the <base>.anno.jls files into the UI. Actually, it loads all <base>.anno*.jls files.

Did I answer all of your questions? Does this make sense?

lukGWF · July 16, 2024, 10:44pm

Hi @mliberty,
Thanks for the prompt informations.
I have done like this

backup info (sources, signals, user_data) from file
write user_data to file with backuped info (if not, the information was lost afterwards):

userdata = {}

def _user_data_cbk_backup(chunk_meta, data):
    suffix = ''
    if len(data) > 64:
        data = data[:64]
        suffix = '...'
    s = f'    {chunk_meta}: {data}{suffix}'
    print(s)
    global userdata
    userdata = {'chunk_meta': chunk_meta, 'data': data, 'suffix': suffix}

def jlsReadNotesFromFile(filename) -> dict:
    """
    Read notes from jls file.
    """
    notes = ""
    if not os.path.isfile(filename):
        return -1
    else:
        try:
            with Reader(filename) as r:
                r.user_data(_user_data_cbk_backup)
                fileinfo = {"sources": r.sources, "signals": r.signals, "user_data": ""}
                # user data
                if "data" in userdata.keys():
                    fileinfo["user_data"] = userdata['data'].decode('utf-8')

        except Exception as e:
            print(e)
            return -1
    return fileinfo


def jlsAddNotesToFile(filename):
           try:
            #data = bytes(range(256))
            fileinfo = jlsReadNotesFromFile(filename)
            with Writer(filename) as wr:
                for src in fileinfo["sources"]:
                    wr.source_def_from_struct(src['source_id'], src)
                for sig in fileinfo["signals"]:
                    wr.source_def_from_struct(sig['signal_id'], sig)
                wr.user_data(0x0000, bytes(notes, 'utf-8'))
                #wr.annotation(1, 1, AnnotationType.USER, 0, data)
        except Exception as e:
            print(e)
        return 0

jlsAddNotesToFile(testfile1.jls)

exception is thrown on first source iteration:

  File "pyjls/binding.pyx", line 341, in pyjls.binding.Writer.source_def_from_struct
AttributeError: 'int' object has no attribute 'source_id'

metadata before (jls taken using record):

{'sources': {0: SourceDef(source_id=0, name='global_annotation_source', vendor='jls', model='-', version='1.0.0', serial_number='-'), 1: SourceDef(source_id=1, name='JS220-002329', vendor='Jetperch', model='JS220', version='', serial_number='002329')}, 'signals': {0: SignalDef(signal_id=0, source_id=0, signal_type=1, data_type=8196, sample_rate=0, samples_per_data=16, sample_decimate_factor=16, entries_per_summary=10, summary_decimate_factor=10, annotation_decimate_factor=100, utc_decimate_factor=100, sample_id_offset=0, name='global_annotation_signal', units='', length=0), 1: SignalDef(signal_id=1, source_id=1, signal_type=0, data_type=8196, sample_rate=1000000, samples_per_data=8192, sample_decimate_factor=128, entries_per_summary=640, summary_decimate_factor=20, annotation_decimate_factor=100, utc_decimate_factor=100, sample_id_offset=20667415523, name='current', units='A', length=991494)}, 'user_data': ''}

after:

{'sources': {0: SourceDef(source_id=0, name='global_annotation_source', vendor='jls', model='-', version='1.0.0', serial_number='-')}, 'signals': {0: SignalDef(signal_id=0, source_id=0, signal_type=1, data_type=8196, sample_rate=0, samples_per_data=16, sample_decimate_factor=16, entries_per_summary=10, summary_decimate_factor=10, annotation_decimate_factor=100, utc_decimate_factor=100, sample_id_offset=0, name='global_annotation_signal', units='', length=0)}, 'user_data': 'Test notes from UT to a JLS file'}

I did get the info from: Python API — pyjls 0.9.6 documentation but it seems not to work properly.
BR

Lukas

mliberty · July 16, 2024, 10:58pm

Hi @lukGWF - I am a little confused as to what you are trying to do. Are you trying to append notes to a file? Once you close a JLS file, it is closed. The JLS file format intentionally does not support open for append. If you want to write additional data to a JLS file, you must do it before closing the Writer.

Are the notes / information you want to add available before you close the JLS Writer?

lukGWF · July 17, 2024, 5:49am

Hi @mliberty,
Now I understand what you have meant, when it’s closed it’s closed. Hm, I use ‘python -m pyjoulescope_driver record’ to capture.

mliberty · July 17, 2024, 11:58am

Hi @lukGWF - The pyjoulescope_driver record entry point does not support adding any custom notes.

I see a few options:

Save a separate file that contains your notes and additional data. For example, if the JLS file is named <base>.jls, you could save your data to <base>.dat.
Create your own JLS writer implementation that does whatever you want. You can use
pyjoulescope_driver.entry_points.record and pyjoulescope_driver.record as an example.
Hack the JLS file to append the data. Although this is unsupported, it should be possible. The JLS raw API does support append mode. It uses this for file repair.

lukGWF · July 18, 2024, 5:41pm

Hi @mliberty,
Thanks for your support.

I have encoded the information in the filename.

Could this be a feature request for the record?

mliberty · July 18, 2024, 6:02pm

Hi @lukGWF - For the record entry point, I can see adding the ability to specify a short string on the command line that would be added as a note. I don’t see any value in adding a note at close since it would require an interactive prompt that we try to avoid in the command-line utilities.

For the record module, you can do whatever you want today by accessing the Writer instance as the _wr member. Not great, but possible.

What exactly do you have in mind for the feature request?

lukGWF · October 15, 2024, 9:51am

Hi,
Your proposal sounds good to add a short string. I had, as already mentioned, coded the Joulescope input voltage into the filename, the bad thing is, that when having a date-time prefix + a notice + the voltage information the filenames get very long.