Big fan of the joulescope. Was trying to use it to collect large amounts of data over time and an issue was nans. I have read a couple of the other posts here concerning nans. I was able to filter nans out of my data using numpy.nanmean, but thought maybe it would be a cool feature if the joulescope library had function versions that ignored nans as well.
Hi @joulescopeluver - Joulescopes use NaN to indicate that a sample was not received from the instrument. This occurs most often when your host computer goes off and does something else rather than servicing the Joulescope sample stream over USB. Dropped samples can degrade accuracy. Very occasional sample drops usually do not affect overal accuracy significantly.
Using numpy.nanmean(x) or x[numpy.isfinite(x)] are both valid ways of ignoring NaNs.
Are you seeing a lot of NaN samples with your setup?
The nans are very random, but I would give a very rough estimate of 1/250 samples are nans.
Hi @joulescopeluver - I would classify 1/250 missing samples as significant data loss.
The Joulescope UI Waveform widget displays missing samples a faint vertical bars. Are you seeing lots of faint vertical bars in the UI? If you want, let the UI run for a while, then select Help → Report Issue. We can take a look at the log. Just place a link to this forum thread in the description when you submit the issue.
Here are some things to consider when tracking down sample drops:
- USB is a shared bus. Any other devices on the same USB root hub can cause the host to not service other devices. Full-speed and slow-speed devices that transfer lots of data are particularly problematic.
- If you are using your own Python code to capture data, ensure that your code does not block or sleep in any joulescope or pyjoulescope_driver callback.
- Disable power management on your host computer. Some computers, especially laptops running on battery, throttle peripheral servicing very aggressively.
My issue seems to be that I have probably 10 things plugged into USB, I was just saying it would be a cool feature for the library to filter nans out automatically.
Hi @joulescopeluver - Ideally, you would never see a NaN sample. I took a quick look at the documentation for pyjoulescope and pyjoulescope_driver, and I did not see a mention of NaN’s. We can definitely do a better job on the docs.
The default behavior needs NaN’s to keep sample time aligned. We can consider adding an optional argurment to some functions to filter NaNs for you. Which package and functions/methods are you using?
The only function I am using is read for my JS220.