This is the second part of my short series on my experiences with Elm while
Check out my part one
for some background info.
Fuzz testing in Elm
I will assume that you are familiar with the general concept of fuzz testing aka
generative testing aka property based testing.
If you are not familiar with it yet, you can get a nice intro from the
elm-test readme or in case you
have access to frontend-masters please take a look at the fuzz testing part of
It is short and simple and is enough to get you started.
Fuzz testing all the time
Now that the background should be clear. There is only one thing I want to mention
in this post: Use fuzz testing, ideally in every project!
Even if you think it is not worth to add it to a certain project.
I don’t mean use it for everything but prefer it over unit testing as the goto tool
and you will quickly be surprised by its value.
My experience in Elm
I added tests and fuzz testing to Carna just to get used to it. Then surprisingly
my tests failed for an unexpected input. You can find the actual test that failed
My tests are really rudimentary but they still found multiple bugs that where
Unexpected String.toInt behavior
In Carna I am asking for numeric values like weight or height and parse them to Int.
(Now they are Float, but that is not important here)
Thus the value is represented as a
Result String Int. That is also what is tested
at the line above. Then my tests failed for the input “+”. The reason:
> String.toInt "+"
Ok NaN : Result.Result String Int
I think I would have never thought about using “+” or “-“ as a input value that
would cause an
Ok result. Especially I would not have thought that the state
Ok NaN was even possible.
What I was expected is something that happens when you use
> String.toFloat "+"
Err "could not convert string '+' to a Float" : Result.Result String Float
This bug was not severe because
toString NaN just resulted in my input fields
having the content “NaN” if you enter “+” which is not is better than crashing
but not what I wanted.
I asked about the behavior in the elm slack and
got quick and helpful feedback there as well as on twitter. So thanks to the
great, friendly and helpful community. ❤️❤️❤️
I learned that the current state is related to the current behavior of toInt and
be fixed in Elm 0.19, which is great news!
This was only the first issue in a list of bugs my fuzz tests found.
Afterwards I switch to regex matches instead of relying on toInt never creating
invalid Ok states, and I forget to allow decimal points for floats, because
I used the same pattern for Int and Float validation, but my fuzz tests found
that quickly. And there were more, but I think my point is clear.
Combine with test watch mode
I realized that elm-test watch mode is great to have a higher number of generated
tests cases. By default 100 inputs are generated per fuzz-invocation.
You can change that value with the
--fuzz option of elm-test. However this can
quickly become annoying if you are executing your tests manually and wait for the
result. However in CI many people already use higher values. But you can also
increase the number locally when combining it with the recently added watch mode
that you can start with
--watch. Because the tests start immediately after saving
the file so they already run when you switch to the console (if you do that at all).
I use this line here for local development:
elm-test --watch --fuzz 10000
Since the numbers the fuzzers are biased towards values that are typical error
cases higher values helped my to find more edge cases quicker.
For more details take a look at elm-test Fuzz.frequency.
I guess if you have way more fuzz tests you might go down to 2000 or 1000
but this is still much more than 100 and it did not annoy me at all yet.
I really think everybody should consider adding fuzz testing to their Elm projects.
Even in my case of a private side project it was very valuable because there are
not only edge cases that you are not aware of, but you might also find bugs
in libraries or tools you use. This was for example the case for riak which had
a high priority bug discovered via generative testing.
In addition I also recommend this article from basho
for further reading and do not forget to increase your
--fuzz value in case
you use watch mode and on CI.