test_solve - hypothesis.errors.InvalidArgument: min_size=8_325 is larger than Hypothesis is designed to handle while generating 'x2'

[asmeurer]

__________________________________ test_solve __________________________________

    @pytest.mark.xp_extension('linalg')
>   @given(*solve_args())

array_api_tests/test_linalg.py:616: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <[AttributeError("'ListStrategy' object has no attribute 'element_strategy'") raised in repr()] ListStrategy object at 0x7eff05717450>
elements = builds(complex, FloatStrategy(min_value=-1.8446742974197924e+19, max_value=1.8446742974197924e+19, allow_nan=False, sm...9, max_value=1.8446742974197924e+19, allow_nan=False, smallest_nonzero_magnitude=1.1754943508222875e-38).map(downcast))
min_size = 8325, max_size = 8325

    def __init__(self, elements, min_size=0, max_size=float("inf")):
        super().__init__()
        self.min_size = min_size or 0
        self.max_size = max_size if max_size is not None else float("inf")
        assert 0 <= self.min_size <= self.max_size
        if min_size > BUFFER_SIZE:
>           raise InvalidArgument(
                f"min_size={min_size:_d} is larger than Hypothesis is designed to handle"
            )
E           hypothesis.errors.InvalidArgument: min_size=8_325 is larger than Hypothesis is designed to handle
E           while generating 'x2' from one_of(floating_dtypes(), complex_dtypes()).flatmap(lambda d: arrays(d, *args, elements=elements, **kwargs))
E           
E           You can reproduce this example by temporarily adding @reproduce_failure('6.98.13', b'AXicU2VlYGBkZEAHjCCoCwAFRABd') as a decorator on your test case

/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/hypothesis/strategies/_internal/collections.py:147: InvalidArgument

I'm not completely clear from the information given what the issue is. I don't see any issues with the arrays generated for test_solve https://github.com/data-apis/array-api-tests/blob/master/array_api_tests/test_linalg.py#L592, nor have I seen this error before.

[asmeurer]

Was able to reproduce in a debugger, but this is deep in hypothesis code. This looks like it's probably a hypothesis bug? Most likely one introduced since 6.97.1, which was the version I was previously using locally.

It seems to be having an issue with generating a list of 8000 elements in order to generate an array, but we generate arrays of that size all the time (our MAX_ARRAY_SIZE is 10000).

If this really is an issue with my strategy and not a hypothesis bug, there is a meta-hypothesis issue in that the error is completely incoherent.

[asmeurer]

Nothing stands out to me in the changelog, but I ran tens of thousands of examples in 6.97.1 and didn't get this error.

[asmeurer]

Also confirmed that @reproduce_failure('6.97.1', b'AXicU2VlYGBkZEAHjCCoCwAFRABd') does not reproduce the failure.

[asmeurer]

By the way, to reproduce this you may need to run against data-apis/array-api-compat#93 (numpy backend)

[asmeurer]

Seems like this was a recent change in hypothesis, but maybe the arrays() strategy was never updated HypothesisWorks/hypothesis#3896? I'm still confused why my solve strategy only is hitting this. CC @Zac-HD

[asmeurer]

Can you please share the code in question, preferably a minimal reproducing example?

So you see what I mean about the error message being unhelpful? :)

To reproduce, checkout this branch data-apis/array-api-compat#93, then run

PYTHONPATH=path/to/array-api-compat ARRAY_API_TESTS_MODULE=array_api_compat.numpy pytest array_api_tests/test_linalg.py -k solve

in this repo.

I linked to the relevant strategy above. I don't think there's any unique or anything. Just generating two arrays with related shapes.

[Zac-HD]

I've now spent more than an hour unsuccesfully trying to observe this error.

Can you give me a single-file reproducer with minimal dependencies?

[asmeurer]

I'll see what I can do on Monday. There are definitely a lot of moving parts here. It might be best to wait for @honno. I believe he is on vacation until late next week.

[asmeurer]

For me,

ARRAY_API_TESTS_MODULE=numpy.array_api pytest array_api_tests/test_linalg.py -k solve --max-examples=1000

reproduces the error. No need to use array-api-compat. This time the error looked like

_______________________________________________________________________________ test_solve ________________________________________________________________________________

    @pytest.mark.xp_extension('linalg')
>   @given(*solve_args())

array_api_tests/test_linalg.py:619:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <[AttributeError("'ListStrategy' object has no attribute 'element_strategy'") raised in repr()] ListStrategy object at 0x16a1fe710>
elements = builds(complex, FloatStrategy(min_value=-1.8446742974197924e+19, max_value=1.8446742974197924e+19, allow_nan=False, sm...9, max_value=1.8446742974197924e+19, allow_nan=False, smallest_nonzero_magnitude=1.1754943508222875e-38).map(downcast))
min_size = 8288, max_size = 8288

    def __init__(self, elements, min_size=0, max_size=float("inf")):
        super().__init__()
        self.min_size = min_size or 0
        self.max_size = max_size if max_size is not None else float("inf")
        assert 0 <= self.min_size <= self.max_size
        if min_size > BUFFER_SIZE:
>           raise InvalidArgument(
                f"min_size={min_size:_d} is larger than Hypothesis is designed to handle"
            )
E           hypothesis.errors.InvalidArgument: min_size=8_288 is larger than Hypothesis is designed to handle
E           while generating 'x2' from shared(sampled_from([(dtype('float32'), dtype('float32')), (dtype('float64'), dtype('float64')), (dtype('complex64'), dtype('complex64')), (dtype('complex128'), dtype('complex128')), (dtype('float32'), dtype('float64')), (dtype('float64'), dtype('float32')), (dtype('complex64'), dtype('complex128')), (dtype('complex128'), dtype('complex64'))])).map(lambda pair: pair[1]).flatmap(lambda d: arrays(d, *args, elements=elements, **kwargs))

../../miniconda3/envs/array-apis/lib/python3.11/site-packages/hypothesis/strategies/_internal/collections.py:147: InvalidArgument

This is with hypothesis 6.98.17

[Zac-HD]

If this really is an issue with my strategy and not a hypothesis bug, there is a meta-hypothesis issue in that the error is completely incoherent.

The reason I'm asking for a single-file repro is to have a small enough volume of code that I can tell why you're hitting this error. I'm fairly confident that it's inside your strategy, but knowing how will help improve the message etc.

...and while staring at that message again, I think part of the problem is that we're raising an exception before assigning all the attributes used in the __repr__, such that the exception message crashes out in a rather unhelpful way. I'm away from my dev setup this week, but if you want to move that if-clause from the middle to the end of ListStrategy.__init__ we should get a more informative message.

[Zac-HD]

Try installing Hypothesis from HypothesisWorks/hypothesis@master...Zac-HD:hypothesis:fix-lists-error:

pip install git+https://github.com/Zac-HD/hypothesis@fix-lists-error#subdirectory=hypothesis-python

[asmeurer]

data-apis/array-api-compat#102

[asmeurer]

What an I missing here? It seems to have installed your branch, https://github.com/data-apis/array-api-compat/actions/runs/8208903123/job/22453311483?pr=102#step:5:61, but the traceback doesn't look it has your update with the check moved lower in __init__.

__________________________________ test_solve __________________________________

    @pytest.mark.xp_extension('linalg')
>   @given(*solve_args())

array_api_tests/test_linalg.py:619: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <[AttributeError("'ListStrategy' object has no attribute 'element_strategy'") raised in repr()] ListStrategy object at 0x7f8b772cf610>
elements = builds(complex, FloatStrategy(min_value=-1.3407807929942596e+154, max_value=1.3407807929942596e+154, allow_nan=False, ...29942596e+154, max_value=1.3407807929942596e+154, allow_nan=False, smallest_nonzero_magnitude=2.2250738585072014e-308))
min_size = 8400, max_size = 8400

    def __init__(self, elements, min_size=0, max_size=float("inf")):
        super().__init__()
        self.min_size = min_size or 0
        self.max_size = max_size if max_size is not None else float("inf")
        assert 0 <= self.min_size <= self.max_size
        if min_size > BUFFER_SIZE:
>           raise InvalidArgument(
                f"min_size={min_size:_d} is larger than Hypothesis is designed to handle"
            )
E           hypothesis.errors.InvalidArgument: min_size=8_400 is larger than Hypothesis is designed to handle
E           while generating 'x2' from shared(sampled_from([(<class 'numpy.float32'>, <class 'numpy.float32'>), (<class 'numpy.float64'>, <class 'numpy.float64'>), (<class 'numpy.complex64'>, <class 'numpy.complex64'>), (<class 'numpy.complex128'>, <class 'numpy.complex128'>), (<class 'numpy.float32'>, <class 'numpy.float64'>), (<class 'numpy.float64'>, <class 'numpy.float32'>), (<class 'numpy.complex64'>, <class 'numpy.complex128'>), (<class 'numpy.complex128'>, <class 'numpy.complex64'>)])).map(lambda pair: pair[1]).flatmap(lambda d: arrays(d, *args, elements=elements, **kwargs))
E           
E           You can reproduce this example by temporarily adding @reproduce_failure('6.98.17', b'AXic42dkAEIQwczCCAIMDAyMbCxAgpGZARkcYMAEWMQYsTJHhXEIN4g8Ml38lIGBONXz/0PAoHXgqPCo8EgVnjGtgSpmMyoDAFmrEfM=') as a decorator on your test case

/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/hypothesis/strategies/_internal/collections.py:147: InvalidArgument
---------------------------------- Hypothesis ----------------------------------
WARNING: Hypothesis has spent more than five minutes working to shrink a failing example, and stopped because it is making very slow progress.  When you re-run your tests, shrinking will resume and may take this long before aborting again.
PLEASE REPORT THIS if you can provide a reproducing example, so that we can improve shrinking performance for everyone.

[Zac-HD]

I think it's actually using this earlier install: https://github.com/data-apis/array-api-compat/actions/runs/8208903123/job/22453311483?pr=102#step:5:47 - maybe try uninstalling first, then installing from github?

In any case I've now released those changes in Hypothesis 6.98.18, so you can install that by version.

[honno]

I think it's actually using this earlier install: https://github.com/data-apis/array-api-compat/actions/runs/8208903123/job/22453311483?pr=102#step:5:47 - maybe try uninstalling first, then installing from github?

In any case I've now released those changes in Hypothesis 6.98.18, so you can install that by version.

Brill very handy @Zac-HD 😁

If this really is an issue with my strategy and not a hypothesis bug, there is a meta-hypothesis issue in that the error is completely incoherent.

So yeah the error could of been more specific but it looks like an issue with the solve_args() strategy factory, which should be resolved by #242.

[asmeurer]

Here's the error message from the latest version of hypothesis:

_______________________________________________________________________________ test_solve ________________________________________________________________________________

    @pytest.mark.xp_extension('linalg')
>   @given(*solve_args())

array_api_tests/test_linalg.py:619:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = ListStrategy(builds(complex, FloatStrategy(min_value=-1.8446742974197924e+19, max_value=1.8446742974197924e+19, allow_...19, allow_nan=False, smallest_nonzero_magnitude=1.1754943508222875e-38).map(downcast)), min_size=8_288, max_size=8_288)
elements = builds(complex, FloatStrategy(min_value=-1.8446742974197924e+19, max_value=1.8446742974197924e+19, allow_nan=False, sm...9, max_value=1.8446742974197924e+19, allow_nan=False, smallest_nonzero_magnitude=1.1754943508222875e-38).map(downcast))
min_size = 8288, max_size = 8288

    def __init__(self, elements, min_size=0, max_size=float("inf")):
        super().__init__()
        self.min_size = min_size or 0
        self.max_size = max_size if max_size is not None else float("inf")
        assert 0 <= self.min_size <= self.max_size
        self.average_size = min(
            max(self.min_size * 2, self.min_size + 5),
            0.5 * (self.min_size + self.max_size),
        )
        self.element_strategy = elements
        if min_size > BUFFER_SIZE:
>           raise InvalidArgument(
                f"{self!r} can never generate an example, because min_size is larger "
                "than Hypothesis supports.  Including it is at best slowing down your "
                "tests for no benefit; at worst making them fail (maybe flakily) with "
                "a HealthCheck error."
            )
E           hypothesis.errors.InvalidArgument: ListStrategy(builds(complex, FloatStrategy(min_value=-1.8446742974197924e+19, max_value=1.8446742974197924e+19, allow_nan=False, smallest_nonzero_magnitude=1.1754943508222875e-38).map(downcast), FloatStrategy(min_value=-1.8446742974197924e+19, max_value=1.8446742974197924e+19, allow_nan=False, smallest_nonzero_magnitude=1.1754943508222875e-38).map(downcast)), min_size=8_288, max_size=8_288) can never generate an example, because min_size is larger than Hypothesis supports.  Including it is at best slowing down your tests for no benefit; at worst making them fail (maybe flakily) with a HealthCheck error.
E           while generating 'x2' from shared(sampled_from([(dtype('float32'), dtype('float32')), (dtype('float64'), dtype('float64')), (dtype('complex64'), dtype('complex64')), (dtype('complex128'), dtype('complex128')), (dtype('float32'), dtype('float64')), (dtype('float64'), dtype('float32')), (dtype('complex64'), dtype('complex128')), (dtype('complex128'), dtype('complex64'))])).map(lambda pair: pair[1]).flatmap(lambda d: arrays(d, *args, elements=elements, **kwargs))

../../miniconda3/envs/array-apis/lib/python3.11/site-packages/hypothesis/strategies/_internal/collections.py:152: InvalidArgument

[Zac-HD]

ListStrategy(builds(complex, ...), min_size=8_288, max_size=8_288) can never generate an example, because min_size is larger than Hypothesis supports.

yep, that's your problem.