Evaluating large measurements, afterwards and during measurement

Hello,
i where doing a three day 10kHz measurement that resulted in deed in a 35GB JouleScope Log file. If I try to open it, it takes a very long time until I get the first rougth view of the measurement.
And If I try to zoom in very close, so I have only 1 second on my screen, it takes a lot of time.

Is there a way to slice the measurement file in peaces, that could be loaded faster?
Is there a way to configure a sliced logging, so that after a certain amount of data a new file is started?

And one other thing, when I do such a long measurement, it sometimes would be good to freeze the screen and then operate on the buffer, still running the logging in background further. Is such a feature planned?

best regrads and thanks in advance
ps_Water

Hi @ps_Water, and welcome to the forum!

Your Joulescope can certainly capture a lot of data. The log file format has been designed to allow for terabytes, but that does sacrifice some performance. Are you recording to a traditional spinning Hard Disk Drive (HDD) or a Solid State Drive (SSD)? If you are using a HDD, an SSD will be much faster, not too expensive, and 36GB is not too big for an SSD.

We do not have a feature to slice a capture into multiple files, and we do not have this feature planned. Unfortunately, slicing the capture would likely create more problems for many Joulescope customers. However, when you find data of interest, you can export that data to its own file. Here’s how:

  1. Right-click on the x-axis
  2. Select Annotations → Dual Markers
  3. Left-click and drag to position the markers
  4. Right-click on one of the dual markers
  5. Select Export data

We do have an item on our list to continue to improve JLS read performance. The existing JLS read code is in Python, and which we can likely substantially improve. If anyone reading this agrees that this should be a priority, comment below!

We do not have a feature to freeze the screen and navigate during a capture. Unfortunately, this feature would be difficult to implement, given the existing architecture. I can see how this would be valuable. The closest solution I can think of is to use dual markers to select the region and export it while the capture is still running. You can then open another Joulescope UI instance and load the JLS file. We did not design for this use case. However, I just tried it, and it seems to work. Does this work for your case?

Hi @mliberty,

thanks for the quick answer.
To the first element: Sad to hear that nothing is planned in this direction.
I already have a SSD and even with this, the procedure you explained and I already work with before, consumes much patience. Such a feature could be a option in the preferences to have a limit in logging time or in size, when the logging is storing data on the log/disk until a new file starts. So that could also be left unlimited for those how needed it and prefer it.

Is there a python script that could also extract the data, so I could use that for slicing the logging in more files?

To the second element: It depends on the logging and buffering time. For my use case this is hard to grab. But I keep you hint in mind. Thanks for that.

A third thing that would be nice: If in due to the big logging file size the UI needs long computation/loading/operation time, that is not seen anywhere. There could be a status, that something happens in background and the UI is not hang. For my feeling I zoom into the signal and only the few loaded and computed samples are still shown, and when after a long while (1 min) I see the newly loaded sample and therefore more signal details. But no indication during that time. (That also happens if I try a histogram on such data. After 5min I gave up and closed the UI, I don’t see if it is still doing something.)

A fourth thing :slight_smile: : If a logging file is loaded it would make sense to reset or show indication messages within the “single value” windows of the UI, they keep the last value from a valid measurement if the device is also connected on the PC, and at the beginning this is always misleading. (Even with knowing it after sometime, I still get misleaded in the beginning.)

Hopefully, I am following along with the different topics. Let me know if I missed something!

1. Slicing recordings
The right answer here is to make JLS reading faster so that you don’t have to worry about it! We understand that some patience is required with the existing 0.9.1 software for very long captures, and the UI does not give an indication of loading progress.

1a. Tools to process JLS files
While we do not have any automatic slicing tools, you can write your own in Python. I just realized that we do not have an example under the pyjoulescope_examples repo. However, you can find an example in pyjoulescope in the recording entry point.

2. Exploring data while capturing
Understood. The problem here is that we cannot store all data to RAM, so this is challenging technically.

3. JLS read progress
Yes, you are right that we do not display progress on JLS read to update the waveform view. However, you should see it for histograms and all of the Analysis tools, like this:


Do you not see a progress window like this? If you do not, could you post the Joulescope UI version you are using including your platform (Windows, Linux, macOS)?

4. Single Value Display on JLS file load
I just tried this, and I confirm that the Single Value Display is not cleared on JLS file load. I understand how this could be confusing. I created Issue #99.

1 Like

Good Morning @mliberty (It is really morning in Germany).

Thanks a lot again for the quick answers.

  1. (including a) I will have a look.
  2. I understand that this is a very challenging task. I would be happily surprised if such a feature could be anyhow, even with limitations, (not all the current allready run measurement could be seen on freeze), realized in future.
  3. no I don’t get that window, My UI Version is 0.9.2 and the JouleScope driver is 0.9.1. I am using windows 10 pro: 10.0.19041 Build 19041 on an lenovo Laptop with a WD SSD: WDC PC SN730 SDBQNTY-512G-1001.
  4. Thanks for that.

Have a good day
ps_Water

1 Like

You are running the latest and greatest UI on the same version of Win 10 that I am using. I’ll see if I can duplicate this…

Thanks.
One thing I am not sure about. It could be that I did the following:
Load the 32GB measurement with 10kHz sample rate and therefore rough 230000 seconds. Wait unti I had the signals in the scope view. Zoomed to a certain area, about 5min window of the measurement, selected the dual marker, and then choose right click on the dual marker and choose histogram. I did not wait until I had the detailed signals shown in the scope window. I thing it was still loading data in background for the scope. That may influence why the histogram calculation status bar did not show up? .
best regards
ps_Water

1 Like

Thank you for the additional detail. I will capture a 10 kHz sample rate file over the weekend, and see if I can duplicate this.

Hi @mliberty, did you found something?
best regards
ps_Water

Hi @ps_Water and thanks for following up! I did capture the 10 kHz file that is 36 GB and 237642 seconds (2.75 days) long. When I first open it, there is no progress bar. It takes 1:45 minutes to display a reduced-resolution waveform, and 2:30 minutes until the normal resolution waveform.

If I zoom into a 1-second window around 201015 seconds, it takes 41 seconds to display the normal resolution waveform, again with no progress bar. If I add dual markers and then right-click the marker, select Analysis → Histogram, it takes 3 seconds before the “Histogram configuration” window appears. After pressing Ok, the progress dialog box does appear, and the Histogram appears about 3 seconds later.

Now, if I zoom out and then zoom in to another place, drop dual markers, and select Histogram analysis, all before the waveform updates, then the “Histogram configuration” window takes longer to appear, again without a progress bar. If I am impatient, it seems that starting histogram analysis too soon stops the waveform update and then never displays the “Histogram configuration” window. I believe that this successfully duplicates what you reported, and it is repeatable.

For the time being, the workaround is to wait for the waveform to update before running any of the analysis tools. Since the UI does not have a progress bar, you can use “Task Manager” on Windows to see when the CPU utilization for the Joulescope UI drops to 0%.

Based upon the findings, I created the following issues:

  • JLS performance is too slow. See Issue #102.
  • Analysis fails when JLS read updates are in progress. See Issue #103

Thank you for reporting these issues, and the plan is to fix them in the next software release.