ndv live view how-to guide

[jacopoabramo]

• ndv version: latest branch
• Python version: 3.10
• Operating System: Windows 11

Description

I want to stream continously to ndv from multiple channels. I'm starting with a single channel in this case (a synthetic data generator using openwfs). On my system it works (this time I installed pyqt) but the display is a bit wobbly. I'm attaching a video example.

continous-example.mp4

I'm fairly certain that I can do better, maybe change the timer timeout signal or something.

What I Did

Reference script attached

continous_example.py.txt

Requirements

latest ndv branch[pyqt,vispy]
openwfs

[gselzer]

Heh, I was trying to do the same thing a few months back - it's a performance problem, and we should try to rectify this (cc @tlambert03). I think it was some problem around the data wrapper being recreated - when you clear the old images, it removes and repaints the data.

A simple fix is to reuse the data wrapper by providing a buffer to the viewer that you update, instead of providing a new buffer every time. I modified your file to add this:

continous_example.py.txt

Note also that I added a few functions to your SyntheticManager class. You may obtain better performance by calling on_timer when your QTimer times out.

[jacopoabramo]

I was thinking that the solution was relatively simple but I checked the source code for _data and the setter raises an exception, so I thought that maybe I was missing something.

[tlambert03]

yep, just want to chime in here and say the fast streaming in ndv has so far taken a back seat to higher dimensional viewing. however, if we can make some assumptions about the data coming in (for example, if we know that we'll only ever be looking at one frame at a time, and we can always just update the "current" buffer), then I'm sure we can get this to be as fast as vispy itself can be by just directly passing the incoming data buffer to the vispy texture

[jacopoabramo]

yep, just want to chime in here and say the fast streaming in ndv has so far taken a back seat to higher dimensional viewing. however, if we can make some assumptions about the data coming in (for example, if we know that we'll only ever be looking at one frame at a time, and we can always just update the "current" buffer), then I'm sure we can get this to be as fast as vispy itself can be by just directly passing the incoming data buffer to the vispy texture

That would be my approach. In the end I will be forcing the refresh rate around 30/60 Hz for the visualization since I don't need to play cyberpunk. I will be using psygnal so I'm detached from the Qt event loop and the only Qt stuff I'll be using are for widgets. One thing I'm considering is if it's worth to combine msgpack with psygnal in order to have faster transmission but my gut feeling is that I would make it worse.

[jacopoabramo]

@gselzer just a question, is the @ensure_main_thread decorator strictly necessary?

[gselzer]

@gselzer just a question, is the @ensure_main_thread decorator strictly necessary?

Uh, no, it isn't. I saw errors earlier on in my debugging about QObject parents and threads, so I threw it in. That ended up not being the fix, but I forgot to remove it. You can delete it.

[tlambert03]

btw, @jacopoabramo, this is the "simple image preview" widget that we've been using in pymmcore-widgets:

https://github.com/pymmcore-plus/pymmcore-widgets/blob/main/src/pymmcore_widgets/views/_image_widget.py

much faster than ndv for this use case, but doesn't have the other features. so, need to merge the two. but it might give you additional ideas

[jacopoabramo]

I'm tempted to play around with vispy directly at this point but I wanted to explore the option of ndv first - I'm lazy at that, but I also like to support projects by being a customer and providing feedback.

[gselzer]

I'm tempted to play around with vispy directly at this point

@jacopoabramo what are you looking for in using vispy directly?

[jacopoabramo]

@jacopoabramo what are you looking for in using vispy directly?

The most critical thing I would like to have as soon as possible is the first point I mentioned in #60, and that would be viewing datasets adjacently.

[gselzer]

The most critical thing I would like to have as soon as possible is the first point I mentioned in #60, and that would be viewing datasets adjacently.

I mean... if you want to take a stab at it, I think a PR here would be very welcome! 😉

[jacopoabramo]

The most critical thing I would like to have as soon as possible is the first point I mentioned in #60, and that would be viewing datasets adjacently.

I mean... if you want to take a stab at it, I think a PR here would be very welcome! 😉

I mentioned it to @tlambert03 that I would be willing to help though limited I am in my capacity; if you can give me some pointers on where to start looking for that specific feature I can have a look. I was honestly waiting for the MVC V2 API before giving a crack to it; I don't like it too much when too many features confluence into a single branch (past experiences taught me that) but if it's something that can be somehow pushed in the new model I can try to help

[tlambert03]

I was honestly waiting for the MVC V2 API before giving a crack to it

yeah... i was gonna say this might be a good idea. I definitely don't want to squelch any potential enthusiasm for contributions!! Nor do I want to keep important features on hold while we work on a refactor... But i also don't want to frustrate you by having work you spend time on have to be very thoroughly reworked in the near (?🤞) future. If you do try to add something, let's just be very sure to add a test for any public APIs that you may want to ensure keep working after the refactor

thank you!

[jacopoabramo]

I was honestly waiting for the MVC V2 API before giving a crack to it

yeah... i was gonna say this might be a good idea. I definitely don't want to squelch any potential enthusiasm for contributions!! Nor do I want to keep important features on hold while we work on a refactor... But i also don't want to frustrate you by having work you spend time on have to be very thoroughly reworked in the near (?🤞) future. If you do try to add something, let's just be very sure to add a test for any public APIs that you may want to ensure keep working after the refactor

thank you!

I absolutely agree. I had my share of experience with overenthusiastic contributions which I just did not have the heart to block and I don't want to fall victim of the same situation.

[jacopoabramo]

I'm resurrecting this issue since the API changed a lot and now I'm a bit at a loss on how I should change the script to achieve the same result. How should I interact with the viewer in the case I want to embed it in another widget?

[tlambert03]

How should I interact with the viewer in the case I want to embed it in another widget?

see tip in readme:

Tip

To embed the viewer in a broader Qt or wxPython application, you can access the viewer's widget attribute and add it to your layout.

but, good to ping this issue. since A) an example should be added to the docs somewhere and B) I still do want to make a special object that is specifically optimized just for streaming (making sure to make updates as efficient as possible)

[jacopoabramo]

Ok, here's what I got: I stripped down the essentials to just wrap the widget with the following:

import numpy as np
from ndv import ArrayViewer
from qtpy import QtWidgets

class WrapperWidget(QtWidgets.QWidget):

    def __init__(self) -> None:
        super().__init__()
        layout = QtWidgets.QGridLayout()
        self._data = np.ndarray((256, 256), dtype=np.uint8)
        self._viewer = ArrayViewer(self._data)
        layout.addWidget(self._viewer.widget(), 0, 0, 1, 5)

        self.setLayout(layout)

app = QtWidgets.QApplication([])
wrapper = WrapperWidget()

wrapper.show()
app.exec()

When launching this I get the following:

Traceback (most recent call last):
  File "C:\sandbox\python\ndv_wrapper_sandbox.py", line 17, in <module>
    wrapper = WrapperWidget()
              ^^^^^^^^^^^^^^^
  File "C:\sandbox\python\ndv_wrapper_sandbox.py", line 11, in __init__
    self._viewer = ArrayViewer(self._data)
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jacop\miniforge3\envs\mamba-sandbox\Lib\site-packages\ndv\controllers\_array_viewer.py", line 107, in __init__
    self._fully_synchronize_view()
  File "C:\Users\jacop\miniforge3\envs\mamba-sandbox\Lib\site-packages\ndv\controllers\_array_viewer.py", line 251, in _fully_synchronize_view
    self._update_visible_sliders()
  File "C:\Users\jacop\miniforge3\envs\mamba-sandbox\Lib\site-packages\ndv\controllers\_array_viewer.py", line 339, in _update_visible_sliders
    self._view.hide_sliders(hidden_sliders, show_remainder=True)
  File "C:\Users\jacop\miniforge3\envs\mamba-sandbox\Lib\site-packages\ndv\views\_qt\_array_view.py", line 449, in hide_sliders
    self._qwidget.dims_sliders.hide_dimensions(axes_to_hide, show_remainder)
  File "C:\Users\jacop\miniforge3\envs\mamba-sandbox\Lib\site-packages\ndv\views\_qt\_array_view.py", line 230, in hide_dimensions
    layout.setRowVisible(slider, False)
    ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'QFormLayout' object has no attribute 'setRowVisible'

Fairly certain I'm doing something wrong.

EDIT: I don't think it is related to this issue; if this behavior is new to you guys I can open a new issue

[tlambert03]

AttributeError: 'QFormLayout' object has no attribute 'setRowVisible'

setRowVisible was added in qt 6.4. Make sure to follow the installation instructions here: https://pyapp-kit.github.io/ndv/latest/install/

using the [pyqt] extra as explained there will make sure you have a Qt in the compatible range

[jacopoabramo]

That's what I did: at the time of posting the snippet above PyQt version was 6.8; I rolled back to PyQt 6.5 but the error still occurs

Could be another of Windows shenanigans?

[tlambert03]

please paste your environment: pip list

[jacopoabramo]

Ok, nevermind. I was working on a very polluted environment and there were some crashing dependencies apparently. I created a fresh environment and it works fine.

I can probably do a small PR and add this snippet as an example on how to embed ndv in a Qt widget if this is informative

[tlambert03]

sure! I can imagine a new "Recipes" section in the docs? with a section called "embedding in a broader application" or something?

[jacopoabramo]

Got it, I'll work on it