Test-case reducers are underappreciated debugging tools
Posted by ltratt 12 hours ago
Comments
Comment by nnunley 55 minutes ago
I'd love to get some feedback if anyone's interested.
Comment by skybrian 3 hours ago
Comment by Jtsummers 2 hours ago
Comment by akshayshah 2 hours ago
Comment by macintux 2 hours ago
Comment by mrkeen 3 hours ago
The article's approach seems super ad-hoc, leaving you to have to think hard, do all the work, and make all the mistakes.
If you were to go down the other path, you might try dividing and conquering the problem. An arbitrary Pair<A,B> is trivially constructed from an arbitrary A and an arbitrary B. So if you can generate a string, and a number, you could generate a User full of number and string fields. If your generate function accepts a number describing how complex a string to make, then you can also choose how complicated to make your User. That's all shrinking needs to be. Repeatedly trying smaller Ns while the problem still happens (the problem being one of your unit tests - not an additional "interestingness" test you need to write.)
You'll probably way more likely to hit boundary cases by using the structure of the input and making interesting variations that way, rather than hoping you can permute the right bytes from the CLI.
Comment by sigbottle 3 hours ago
On one project, through a variety of circumstances, dead code elimination was straight up not working, but we wanted to show the theoretical improvement of some approach - but we couldn't figure out why at the moment (we did spend a whole week chasing down the root cause after - maybe worth in hindsight...).
We were doing it by hand at one point, but someone suggested using CReduce for shrinking the code. Definitely was an interesting test-iterate loop...
Comment by hungryhobbit 3 hours ago
I'm not sure if that's an article failure (that I didn't want to read a whole ton of text and C code details), or a success (as it got me interested in the topic). I guess both?
Comment by Jtsummers 2 hours ago
It's answered pretty early on:
>> Test-case reducers try to reduce the length of an input
If that still doesn't answer the question, try this extension:
>> Test-case reducers try to reduce the length of an [error causing or interesting] input