Discussion:
selectively running unittest functions
(too old to reply)
Daniel Davidson
2013-10-25 12:51:32 UTC
Permalink
I've been through this thread:
http://forum.dlang.org/post/mailman.1454.1369104411.4724.digitalmars-***@puremagic.com

I would like the named unittests, as describe there, but I don't
think it is in the works (if I'm wrong please let me know). So I
took an approach outlined here
(http://www.reddit.com/r/programming/comments/1edih2/dconf_2013_day_1_talk_4_writing_testable_code_in/c9zg3ry)
and created a minimal, simple scheme.

The support file is here: http://pastebin.com/tU9BuS3G

To use it:
- import the support file
- mixin an init routine `mixin UTInit!__MODULE__;` to your module
- Instead of using unittest blocks (or in addition) put code in
annotated functions.
`@UT void testFoo() { .... }


To run all tests:
`rdmd -version=UT --main program`
To filter by module (using regex)
`rdmd -version=UT --main program -m 'foo.*'`
To filter by test function (using regex)
`rdmd -version=UT --main program -t 'foo.*'`

I think the approach makes sense, but I am looking any
comments/criticisms on what may be missing or misguided.

The downsides as I see it:
- it is not standard
- it comes with a version, UT. So you can/should wrap all tests
in version(UT) to ensure they never make it to production
- all test functions must be at module level. Not sure if there
are any ramifications?

The pluses:
- filtering of test by module and function name
- it works along side of unittest. It can also work with others
like specd which is a nice way to describe tests and know when
things fail the location.


Thanks
Dan
Dicebot
2013-10-25 13:04:02 UTC
Permalink
I strictly believe any unittest enhancing library must be built
on top of existing unittest blocks using __traits(getUnittest)
and be 100% compatible with normal `-unittest` mode
Daniel Davidson
2013-10-25 13:23:45 UTC
Permalink
Post by Dicebot
I strictly believe any unittest enhancing library must be built
on top of existing unittest blocks using __traits(getUnittest)
and be 100% compatible with normal `-unittest` mode
I don't disagree. What exactly does that mean and what would it
look like?

I'm not familiar with __traits(getUnittest). Is that something
that could/should be used in what I'm doing?

What I'm missing, and apparently others in the original thread,
is a way to run tests selectively. It is difficult to do with
unittest because they are not named. If there were a way to
annotate the test and pull them out that way it would be great.
Can it be done?
Dicebot
2013-10-25 13:30:53 UTC
Permalink
Post by Daniel Davidson
What I'm missing, and apparently others in the original thread,
is a way to run tests selectively. It is difficult to do with
unittest because they are not named. If there were a way to
annotate the test and pull them out that way it would be great.
Can it be done?
You can completely re-implement default test by using runtime
hook http://wiki.dlang.org/Runtime_Hooks ("_d_unittest" should do
AFAIK)

In that runner you can manually get all unittest blocks in the
program as functions using __traits(getUnittest). One can make
any custom decisions for running specific unittest blocks based
on User-Defined Attributes attached to it.

Benefit of this approach is that the very same tests remain
runnable in traditional out-of-the-box way if you don't use that
library runner.

I remember Jacob Carlborg doing some experiments in that
direction but did not track any further progress.
Dicebot
2013-10-25 13:35:41 UTC
Permalink
Post by Dicebot
You can completely re-implement default test by using runtime
hook http://wiki.dlang.org/Runtime_Hooks ("_d_unittest" should
do AFAIK)
*default test runner
Daniel Davidson
2013-10-25 13:45:28 UTC
Permalink
On Friday, 25 October 2013 at 13:23:46 UTC, Daniel Davidson
Post by Daniel Davidson
What I'm missing, and apparently others in the original
thread, is a way to run tests selectively. It is difficult to
do with unittest because they are not named. If there were a
way to annotate the test and pull them out that way it would
be great. Can it be done?
You can completely re-implement default test by using runtime
hook http://wiki.dlang.org/Runtime_Hooks ("_d_unittest" should
do AFAIK)
Pretty sure this is what I'm doing. Only I did not know you could
UDA a unittest.
In that runner you can manually get all unittest blocks in the
program as functions using __traits(getUnittest). One can make
any custom decisions for running specific unittest blocks based
on User-Defined Attributes attached to it.
Any samples of how to use __traits(getUnittest,...). I might be
able to work as you suggest if I can get this to work.
Benefit of this approach is that the very same tests remain
runnable in traditional out-of-the-box way if you don't use
that library runner.
I remember Jacob Carlborg doing some experiments in that
direction but did not track any further progress.
Dicebot
2013-10-25 14:14:38 UTC
Permalink
This will work starting with 2.064:

module a;

import std.stdio;

struct ID
{
string data;
}

@ID("one")
unittest
{
writeln("1");
}

@ID("two")
unittest
{
writeln("2");
}

void main()
{
import std.typetuple;
alias tests = TypeTuple!( __traits(getUnitTests, a) );

foreach (test; tests)
{
foreach (uda; __traits(getAttributes, test))
{
static if (is(typeof(uda) == ID))
{
if (uda == ID("two"))
test();
}
}
}
}

It will print stuff twice though because I didn't suppress native
unittest runner. Didn't actually dwell deep into runtime hooks so
don't know what is best way to do it (maybe we need one more
enhancement for it)
simendsjo
2013-10-25 14:43:47 UTC
Permalink
Post by Dicebot
module a;
import std.stdio;
struct ID
{
string data;
}
@ID("one")
unittest
{
writeln("1");
}
@ID("two")
unittest
{
writeln("2");
}
void main()
{
import std.typetuple;
alias tests = TypeTuple!( __traits(getUnitTests, a) );
foreach (test; tests)
{
foreach (uda; __traits(getAttributes, test))
{
static if (is(typeof(uda) == ID))
{
if (uda == ID("two"))
test();
}
}
}
}
It will print stuff twice though because I didn't suppress
native unittest runner. Didn't actually dwell deep into runtime
hooks so don't know what is best way to do it (maybe we need
one more enhancement for it)
This is great. I love the unittest blocks and UDAs. This way I
can still have very easy-entry (as in "just write unittest {}")
unittests while gaining the power of unittest frameworks.

It's a hell of a lot better than having some class with
attributes containing methods with attributes, containing
parameters with attributes :) I'm developing RSI when writing
tests in other languages. (In Java I develop RSI just declaring
variables...)
Daniel Davidson
2013-10-25 16:43:21 UTC
Permalink
Ok. I'll keep pressing. Here is an updated version:
http://pastebin.com/g6FWsTkr

The idea is to be able to just import ut, annotate as you have
described and get unit tests run. I want to mixin the equivalent
of your "main" in each module that just pulls in a module
constructors to evaluate what tests are there. The code in the
paste crashes on the call to getUnitests.

If I comment out `alias tests = TypeTuple!(__traits(getUnitTests,
mod));` then I see similar call to `alias members =
TypeTuple!(__traits(allMembers, mod));` work just fine.

Is this the right way to use this? Any pointers on the
segmentation fault?

Thanks
Dan
Daniel Davidson
2013-10-25 22:36:03 UTC
Permalink
Post by Daniel Davidson
http://pastebin.com/g6FWsTkr
The idea is to be able to just import ut, annotate as you have
described and get unit tests run. I want to mixin the
equivalent of your "main" in each module that just pulls in a
module constructors to evaluate what tests are there. The code
in the paste crashes on the call to getUnitests.
If I comment out `alias tests =
TypeTuple!(__traits(getUnitTests, mod));` then I see similar
call to `alias members = TypeTuple!(__traits(allMembers,
mod));` work just fine.
Is this the right way to use this? Any pointers on the
segmentation fault?
Thanks
Dan
Ok, binary reduction has shown the seg fault was due to a debug
import. This code causes a crash for me - just because of the
getUnittests. If I remove the debug import it works. Since I use
debug imports in general, I'm not so sure it is worthwhile to
pursue the named unit test approach at this time.

import std.typetuple;
import std.stdio;
debug import std.datetime;
unittest { writeln("In Test!!"); }
mixin("alias mod = " ~ __MODULE__ ~ ";");
alias tests = TypeTuple!(__traits(getUnitTests, mod));
static this() {
writeln("Done");
}


Thanks
Dan
Dmitry Olshansky
2013-10-26 08:09:20 UTC
Permalink
Post by Daniel Davidson
http://pastebin.com/g6FWsTkr
The idea is to be able to just import ut, annotate as you have
described and get unit tests run. I want to mixin the equivalent of
your "main" in each module that just pulls in a module constructors to
evaluate what tests are there. The code in the paste crashes on the
call to getUnitests.
If I comment out `alias tests = TypeTuple!(__traits(getUnitTests,
mod));` then I see similar call to `alias members =
TypeTuple!(__traits(allMembers, mod));` work just fine.
Is this the right way to use this? Any pointers on the segmentation
fault?
Thanks
Dan
Ok, binary reduction has shown the seg fault was due to a debug import.
It's a bug so please file it in bugzilla. Don't let them go unnoticed ;)
http://d.puremagic.com/issues/
This code causes a crash for me - just because of the getUnittests. If I
remove the debug import it works. Since I use debug imports in general,
I'm not so sure it is worthwhile to pursue the named unit test approach
at this time.
import std.typetuple;
import std.stdio;
debug import std.datetime;
unittest { writeln("In Test!!"); }
mixin("alias mod = " ~ __MODULE__ ~ ";");
alias tests = TypeTuple!(__traits(getUnitTests, mod));
static this() {
writeln("Done");
}
Thanks
Dan
--
Dmitry Olshansky
Daniel Davidson
2013-10-26 10:34:39 UTC
Permalink
On Saturday, 26 October 2013 at 08:09:26 UTC, Dmitry Olshansky
Post by Dmitry Olshansky
On Friday, 25 October 2013 at 16:43:23 UTC, Daniel Davidson
Post by Daniel Davidson
http://pastebin.com/g6FWsTkr
The idea is to be able to just import ut, annotate as you have
described and get unit tests run. I want to mixin the
equivalent of
your "main" in each module that just pulls in a module
constructors to
evaluate what tests are there. The code in the paste crashes
on the
call to getUnitests.
If I comment out `alias tests =
TypeTuple!(__traits(getUnitTests,
mod));` then I see similar call to `alias members =
TypeTuple!(__traits(allMembers, mod));` work just fine.
Is this the right way to use this? Any pointers on the
segmentation
fault?
Thanks
Dan
Ok, binary reduction has shown the seg fault was due to a
debug import.
It's a bug so please file it in bugzilla. Don't let them go
unnoticed ;)
http://d.puremagic.com/issues/
Yes - I have.
Gary Willoughby
2013-10-25 15:51:25 UTC
Permalink
Post by Dicebot
I strictly believe any unittest enhancing library must be built
on top of existing unittest blocks using __traits(getUnittest)
and be 100% compatible with normal `-unittest` mode
I agree, this should be easy to implement. In my unit testing
framework i could add support to run only unit tests of one
particular module but in the future when the above is released it
would be easy to add it on a per test basis.
Gary Willoughby
2013-10-25 15:55:10 UTC
Permalink
For information how to implement your own unit test handler see
the bottom of the report module here:

https://github.com/nomad-software/dunit

From there you can selectively run tests based on some outside
criteria. I think the new trait will open up further
possibilities too.
Dicebot
2013-10-25 15:58:05 UTC
Permalink
Post by Gary Willoughby
For information how to implement your own unit test handler see
https://github.com/nomad-software/dunit
From there you can selectively run tests based on some outside
criteria. I think the new trait will open up further
possibilities too.
After 2.064 is out and good support of attributed test blocks is
added I really recommend to consider separating core modules that
support it into Phobos proposal. Will need to convince some
conservative guys this is actually needed, but I think it is
doable :P
Daniel Davidson
2013-10-26 11:32:58 UTC
Permalink
Here is a working solution:
https://github.com/patefacio/d-help/blob/master/d-help/opmix/ut.d

Currently it only pulls in unittests at the module level. I'm
sure it will work on unittests scoped to structs/classes, I just
need to figure out how to determine if a compile time named
object is an aggregate, as __traits(getUnitTests, foo) will
complain if foo is not a module or aggregate.

With a source a.d and b.d like in:
https://github.com/patefacio/d-help/blob/master/d-help/opmix/examples/ut/
it can pull out by module name and uda test name. It has a -s
summary option as well.

Here is sample output: http://pastebin.com/fYd7k1Kz

Thanks
Dan

Continue reading on narkive:
Loading...