JS220 duration argument not working

@mliberty Trying to work around email block issues.

I followed the instructions and uninstall psutil and deleted the .pyd file and reinstalled. That seemed to work.

When I run the script to capture a 5 second datalog the script runs indefinitely and I have to press crtl-c to terminate it:

When I do terminate the data capture a file is created. The file size is proportional to the amount of time I let the capture run:

But regardless of how short a time I allow it to run, when I open the file in the GUI, it shows a several hundred second long data capture:

The timebase seems off. I’ve attached the code producing the data capture. Its your sample code with a slight modification.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

# Copyright 2020 Jetperch LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Capture samples to JLS files for all connected Joulescopes."""

from joulescope import scan
from joulescope.data_recorder import DataRecorder
import argparse
import os
import sys
import signal
import time
import datetime
import subprocess

def get_parser():
    p = argparse.ArgumentParser(
        description='Read data from Joulescope.')
    p.add_argument('--duration', '-d',
                   type=lambda d: d if d is None else float(d),
                   help='The capture duration in seconds.')
    p.add_argument('--frequency', '-f',
                   help='The sampling frequency in Hz.')


    p.add_argument('--product_tag', '-p',
                   default='NOTAG',
                   help='Product tags are written to the output file for easier access.   If product_tag is R900V5, the output file will be YYYYMMDDSS_R900v5.jls')  
    
    p.add_argument('--command', '-c',
                   type=str,
                   default='',
                   help='Used to embed a command that you want to execute while the datalogger is running.') 

    p.add_argument('--timedelay', '-t',
                   type=float,
                   default=0.0,
                   help='Used to wait X.X seconds before command is executed.') 

    return p


def run():
    return_code = 0
    quit_ = False

    def do_quit(*args, **kwargs):
        nonlocal quit_
        quit_ = 'quit from SIGINT'

    def on_stop(event, message):
        nonlocal quit_
        quit_ = 'quit from stop'

    args = get_parser().parse_args()
    
    signal.signal(signal.SIGINT, do_quit)
    devices = scan(config='auto')
    print(f'Current Joulescopes: {devices}')
    items = []
    if devices:
        try:
            for device in devices:
                if args.frequency:
                    try:
                        device.parameter_set('sampling_frequency', int(args.frequency))
                    except Exception:
                        # bad frequency selected, display warning & exit gracefully
                        freqs = [f[2][0] for f in device.parameters('sampling_frequency').options]
                        print(f'Unsupported frequency selected: {args.frequency}')
                        print(f'Supported frequencies = {freqs}')
                        quit_ = True
                        return_code = -1
                        break
                    # Create a unique report name based off date and time
                datalogger_name = datetime.datetime.now().strftime(f"%Y%m%d_%H%M%S_{args.product_tag}.jls")
                filepath = r'C:\CBAT_Test\Datalogs' + '\\' + datalogger_name        
                print()
                print(filepath)
                print()
                device.open()
                recorder = DataRecorder(filepath,
                                        calibration=device.calibration)
                items.append([device, recorder])
                device.stream_process_register(recorder)

            if not quit_:
                for device, _ in items:
                    device.start(stop_fn=on_stop, duration=args.duration)
                    if args.command:
                        time.sleep(args.timedelay)
                        subprocess.run(args.command)


            print('Capturing data: type CTRL-C to stop')
            while not quit_:
                time.sleep(0.01)
        finally:
            for device, recorder in items:
                try:
                    device.stop()
                    recorder.close()
                    device.close()
                except Exception:
                    print('exception during close')
                    return_code = -2
        return return_code
    else:
        print('No Joulescope Detected.  Did you forgot to plug it in?')
        sys.exit(f"##teamcity[buildStatus status='FAILURE' text='No Joulescope Detected.  Did you forgot to plug it in?']")
        
if __name__ == '__main__':
    run()

Hi @nate and welcome to the Joulescope forum. I have no idea what was getting in the way of the emails, but I also like the forum better than email. It’s easier to keep track, format, and share!

I see two issues.

  1. The new v1 backend’s start method (v1.0.10 link) does not support duration or contiguous_duration. This is why your capture is not automatically stopping.
  2. The JS220 does not yet correctly support downsampling. We are working on this now!

We are close to completing (2). I will also add in (1), which should be a quick addition. I may have something as early as tomorrow, and I will test it using the provided python_datalogger.py code.

Thanks Matt. Look forward to the new release. Backward compatibility is a huge win for our team so thanks for the support on that front. Our team loves Joulescope!

1 Like

HI @Nate - We released a pyjoulescope version 1.0.11 that should fix the issue. I tested it with your script, and it works for me. Here’s how to update:

pip3 install -U joulescope

You should then be able to run the script, and both the --duration and --frequency arguments should work correctly. Let me know if it works for you!

Matt,

A quick test shows the duration argument working, the appropriate file size created, and the time scale accurate. Thanks again for the quick work of it.

1 Like

If you find anything else, please don’t hesitate to let me know!

While we have been testing the API, we have allocated the majority of the testing effort to the Joulescope UI. Any joulescope package features not used by the UI will be less exercised. Thanks for reporting this issue and helping us to improve the API!

We updated all our computers with the latest joulescope package. We attempted to run an overnight test on JS110 hardware that runs for 8 hours. Here is the command:

C:\Python\python.exe C:\CBAT_Test\Python_Datalogger\python_datalogger.py --duration 28800 --frequency 10000 --product_tag CMIUV2_8HOUR_AVG

Only a 1kb file was produced.

We would expect a 4GB file as you can see in the directory listing from 11/9/2022.

Any ideas?

I just tried this out. With --frequency 10000, the code throws an exception:

python python_datalogger.py --duration 10 --frequency 10000 --product_tag hi
Current Joulescopes: [<joulescope.v1.js110.DeviceJs110 object at 0x00000127598F93C0>]

C:\tmp\20221115_091602_hi.jls

Capturing data: type CTRL-C to stop
<joulescope.data_recorder.DataRecorder object at 0x000001275A408DF0> stream_notify() exception
Traceback (most recent call last):
  File "c:\repos\Jetperch\pyjoulescope\joulescope\v1\device.py", line 415, in _stream_process_call
    rv |= bool(fn(*args, **kwargs))
  File "c:\repos\Jetperch\pyjoulescope\joulescope\data_recorder.py", line 205, in stream_notify
    raise ValueError('stream_buffer length too small.  %s > %s' %
ValueError: stream_buffer length too small.  400000 > 300000

If I increase the sampling rate to --frequency 20000, then it works fine without an exception. It looks like DataRecorder requires the stream buffer to contain a full block of samples, which is hard-coded to 400,000. At 10,000 Hz, this is 40 seconds, but the buffer is only 30 seconds by default. So, you should be able to “fix” the problem by increasing the buffer duration.

Try adding this before the datalogger_name = line:

try:
    buffer_duration = max(30, 400000 * 1.25 / int(args.frequency))
    device.parameter_set('buffer_duration', buffer_duration)
except Exception:
    print(f'Could not set buffer duration')
    quit_ = True
    return_code = -1
    break

The prior v0 stream buffer incorrectly ignored “too small” durations and forced a minimum buffer size. The new implementation respects the duration, and allows for smaller buffers, but that exposes this issue. Ideally, the JLS v1 DataRecorder would not require a full block size in the stream buffer. The newer JLS v2 format (see jls_v2_writer) buffers internally and should work correctly with any stream buffer size.

Hey @mliberty ,

Why did you use the 1.25 factor in the calculation above?

Also, is python_datalogger.py a publically available document or is it specific to OP’s code/question?
Thanks,
Samyukta

Hi @sramnath and welcome to the Joulescope forum!

I used the 1.25 as a quick fudge-factor to ensure that the division always rounded up sufficiently. It does not really matter as long as the buffer is longer than 400,000 samples, which is not a burden on any modern host device running python.

Somewhat. The code that @Nate is concerned about is based upon the capture_all.py example script. @Nate modified this example with the modifications posted above.


Hope this helps! If you have additional questions, please feel free to create a new topic.

Thanks @mliberty for such a prompt response.
In your response to @Nate , you mentioned a newer JLS v2 format should work correctly with any stream buffer size.
a. Does that mean that we would not need to add the workaround above while using JLS v2 format?
b. Is this already available to use?
c. If so, how would I install/use it? (joulescope version I’m using = 1.0.15

~/joulescope/Janus/pyjoulescope_examples$ joulescope info
System information
    Python: 3.8.10 (default, Nov 14 2022, 12:59:47) 
[GCC 9.4.0]
    Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.29 (linux)
    Processor: x86_64
    executable: /usr/bin/python3
    frozen: False

joulescope version: 1.0.15
Found 1 connected Joulescope:
    JS220-000402   ctl=1.0.4            sensor=1.0.3

Hi @sramnath - Yes, JLS v2 is ready for write/read from scripts today. The Joulescope UI displays JLS v2 files. JLS v2 is still under development but the implemented API is stable. The most important, unimplemented feature is recovery for improperly closed files (like writing when the program crashes), but it is ready for most purposes. We plan to add the error recovery in the next couple of months.

If you are just trying to record from a single Joulescope, we already have the capture_jls_v2.py example that may work for you.

The pyjls package should be already installed by the joulescope installation. If not, you can run:

pip3 install -U pyjls

Thanks a ton @mliberty . Looks like the capture_jls_v2.py file does not take frequency arguments. What is the frequency used by default / how can I set the frequency used here?

Hi @sramnath - The frequency is the full rate, which is 2 Msps for the JS110 and 1 Msps for the JS220.

If you want to change the sampling rate for capture_jls_v2.py from the command line, you can modify it using the code in capture.py. Here’s how:

  1. Insert catpure.py lines 84-92 before existing line 61.
  2. Insert capture.py lines 42-43 before existing line 41.

Does this work for you?

Yes, it works for me. Thank you! I’m extremely thankful for your prompt responses.

1 Like