Joulescope breaking pyserial on MacOS

Issues
1

Not sure how else to describe it, but importing joulescope breaks pyserial :sweat_smile:

To reproduce, I set up a new environment with Python 3.8.15 and installed pyserial and joulescope

The following script reproduces the issue:

import sys
import serial

test_device = serial.Serial(sys.argv[1])
test_device.write("\n".encode())
test_device.close()
print("OK 1!")

import joulescope

test_device = serial.Serial(sys.argv[1])
test_device.write("\n".encode())
test_device.close()
print("OK 2!")

Here’s the output I get (replace /dev/cu.Bluetooth-Incoming-Port with any serial device):

python test.py /dev/cu.Bluetooth-Incoming-Port
OK 1!
Traceback (most recent call last):
  File "test.py", line 12, in <module>
    test_device.write("\n".encode())
  File "/Users/alvaro/miniconda3-intel/envs/joulescope/lib/python3.8/site-packages/serial/serialposix.py", line 640, in write
    abort, ready, _ = select.select([self.pipe_abort_write_r], [self.fd], [], None)
ValueError: filedescriptor out of range in select()

If I comment out the import joulescope line, the output is as follows:

python test.py /dev/cu.Bluetooth-Incoming-Port
OK 1!
OK 2!

I’m running on MacOS 12.6.2 on an M1 machine, but we’ve had this issue also happen on x64_64 ones.

Here’s the pip freeze output for reference:

pip freeze
certifi==2022.12.7
charset-normalizer==3.0.1
idna==3.4
importlib-metadata==6.0.0
importlib-resources==5.10.2
joulescope==1.0.15
libusb==1.0.26b5
numpy==1.24.1
packaging==23.0
pkg-about==1.0.8
psutil==5.9.4
pyjls==0.4.3
pyjoulescope-driver==1.1.2
pymonocypher==3.1.3.1
pyserial==3.5
python-dateutil==2.8.2
requests==2.28.2
six==1.16.0
tomli==2.0.1
urllib3==1.26.14
zipp==3.11.0
2

I updated the script to print out the file descriptors.

import sys
import serial

test_device = serial.Serial(sys.argv[1])
print("test_device.fileno() = ", test_device.fileno())
test_device.write("\n".encode())
test_device.close()
print("OK 1!")

import joulescope

test_device = serial.Serial(sys.argv[1])
print("test_device.fileno() = ", test_device.fileno())
test_device.write("\n".encode())
test_device.close()
print("OK 2!")

Seems like import joulescope opens a whole lot of files in the process.

test_device.fileno() =  4
OK 1!
test_device.fileno() =  1045
Traceback (most recent call last):
  File "test.py", line 14, in <module>
    test_device.write("\n".encode())
  File "/Users/alvaro/miniconda3-intel/envs/joulescope/lib/python3.8/site-packages/serial/serialposix.py", line 640, in write
    abort, ready, _ = select.select([self.pipe_abort_write_r], [self.fd], [], None)
ValueError: filedescriptor out of range in select()

Looks like pyserial uses select.select() and macos has a 1024 limit on select (from sockets - How to increase filedescriptor's range in python select() - Stack Overflow)

3

I was able to duplicate the problem using your script. I then tried sudo launchctl limit maxfiles 65536 200000, but that didn’t work.

I think that you are right that the process may have too many open files for a single select, but I am not sure why pyserial would provide the joulescope file descriptors, which are actually running on different threads.

I am off to my CM for the remainder of the day, but I’ll see if I can find anything this evening.

4

I have a workaround now. Looks like pyserial does have a way to use select.poll() instead of select.select() by using serial.PosixPollSerial instead of serial.Serial. Unfortunately, this only applies to reads, not writes :man_facepalming:

The only way around the select.select() on the .write() call is to open the port with write_timeout=0. This seems to be enough to get things working, I think :sweat_smile:

import sys
import serial

test_device = serial.PosixPollSerial(sys.argv[1], write_timeout=0)
print("test_device.fileno() = ", test_device.fileno())
test_device.write("\n".encode())
test_device.close()
print("OK 1!")

import joulescope

test_device = serial.PosixPollSerial(sys.argv[1], write_timeout=0)
print("test_device.fileno() = ", test_device.fileno())
test_device.write("\n".encode())
test_device.close()
print("OK 2!")
5

I figured out the root issue. The joulescope_driver libusb backend uses 2 pipes for each possible device to communicate between the upper-level driver and the lower-level driver. Each pipe has 2 file descriptors: one for read and one for write. The joulescope_driver was set to a maximum of 256 devices, and allocated all communication pipes at initialization for a total of 256 * 2 * 2 = 1024 file descriptors. The joulescope_driver also allocates a few other file descriptors.

The solution, at least for now, is to reduce the maximum number of simultaneously supported Joulescope devices connected over USB to 127. This reduces the total number of file descriptors by 516. With this change, the original example above now works. I will be happy to figure out a better solution for the first customer that tries to connect >127 Joulescopes simultaneously to one host computer. I think the current record is approximately 45, so I suspect 127 will be good for quite some time.

The new pyjoulescope_driver 1.1.3 should fix the original issue. To update:

pip3 install -U pyjoulescope_driver

Does this work for you?

6

Thanks Matt! That does solve the problem.

Now that it works we’ve discovered that the extio_status() is not implemented in the new api and when we try JOULESCOPE_BACKEND=0, libusb gets upset :sweat_smile:

7

I took a look at the code. The JS220 extio_status implementation leaves something to be desired:

    def extio_status(self):
        """Read the EXTIO GPI value.
        :return: A dict containing the extio status.  Each key is the status
            item name.  The value is itself a dict with the following keys:
            * name: The status name, which is the same as the top-level key.
            * value: The actual value
            * units: The units, if applicable.
            * format: The recommended formatting string (optional).
        """
        return {}  # todo

And the JS110 is no better:

    def extio_status(self):
        return {}   # todo

JOULESCOPE_BACKEND=0 is tested on Windows, but I am not sure if it was ever tested with libusb (macOS & Linux).

I will take a look at both issues tomorrow.

So that I can best prioritize things, are you using JS110’s, JS220’s, or both?

8

Oh, and if you are just using the JS110 and want to get back to work while I figure things out, you can install the last pyjoulescope release before the JS220 updates:

pip3 install joulescope==0.9.7
9

Yeah, I tried that but maybe the latest libusb doesn’t play nice with it :confused:
I’ve reverted to hardcoding serial numbers (we were using the gpio’s to identify multiple joulescopes in a fixture :sweat_smile:)

10

Ah, then the JOULESCOPE_BACKEND=0 trick won’t work either. At the rate that Apple breaks things innovates, it was hard to keep the package up to date with all the current libusb dynamic libraries. The new joulescope_driver compiles libusb with the same settings as the rest of the code to solve this problem.

I’ll focus on adding back extio_status, which was a definite oversight in our backwards compatibility.

11

Yeah, totally understandable. Our test machine was old enough that it had just the right things and we had to update something else and everything went to hell :joy:

We’re back up and running now with the file descriptor fix and using serial numbers to identify joulescopes instead of extio. :grinning:

12

Hi @alvarop - I added extio_status support to the pyjoulescope v1 implementation. You should be able to update the packages, and everything should work:

pip3 install -U joulescope

As of today, this command should install pyjoulescope_driver 1.1.4 and joulescope 1.0.16.

If you run into further issues with this fix, please post here!

1 Like