Apache OpenOffice (AOO) Bugzilla – Issue 87469
unittesting infrastructure
Last modified: 2013-03-11 15:01:39 UTC
I would like to up-stream the unit testing infrastructure we have in ooo-build so far. The idea is that running unit tests should be as easy as possible - ideally by issuing 'build check' in the module subdirectory which would take care of building that module, and consequently running the tests. To distinguish the tests and the 'real code', we introduced prj/tests.lst which has the same form as build.lst (minus the 1st 'dependency' line'). The subdirs described in the tests.lst are built just in case 'check' is provided to build.pl, and also this 'check' is forwarded to dmake that actually compiles the subdir - so that the executing the tests themselves could be triggered. I'll attach the patch that extends build.pl the way described above - please have a look. If you agree with it, I'll commit to a CWS, and then will do follow-up CWS(es) to up-stream the actual prj/tests.lst's (which are based on the tests that already exist in the OOo) we have so far.
Created attachment 52304 [details] The patch.
@vg, please work with kendy getting this integrated,
I think, you are reinventing the wheel. In the build.lsts we have a "job" field. Usually it's 'nmake' (which triggers dmake in correspondent directory). We can introduce a new job, say 'unit_test' which triggers dmake in the test directory... The trigger can be activated via environment variable or additional build tool switch. Thus we do not need to introduce a new level of complexity and make use of dependency processing... Actually, I do not like the idea of additional dmake switch (the 'check' one), especially when we use separate directory for this purpose. I suspect there is a possibility to solve the problem by writing a proper makefile.mk of the test directory (and add the test directory to build.lst as a normal 'nmake' one)... Both approaches have they pros & contras, but I like them more than just up-stream the existing solution.
vg: The thing is that I'd like to be as close to the usual behavior as possible. By 'usual' I mean behavior that is common in the projects using autocorf/automake and the alike - there 'check' is the target responsible to issue unit tests. In OOo, things get built by running the 'build' command - so the closest resemblance of 'make check' is 'build check' - and that is what I want having working ;-) wrt. the separate tests.lst: Developers should be encouraged to create unit tests; which of course starts with easiness and visibility of usage of the framework for the test. prj/tests.lst is visible, unit_test 'job' hidden in prj/build.lst is not. wrt. the dmake 'switch': For dmake, it is not a switch, it is a normal target that must be implemented in the test's makefile.mk so that it runs the actual test (without that, it just compiles that), like: check: testshl2 blah ugh aoeu This is necessary because not all tests will use testshl2 framework for testing, etc. And running tests should be a separate task from actually building the tests. All in all ;-) - I don't really care how we actually implement it as long as - we have a simple and standard-looking way of issuing them (build check) - it is easy and encouraging for developers to add their tests (so you don't have to read tons of documentation to actually learn about some unit_test trigger in build.lst [which is cryptic enough by itself already ;-)])
vg: we have to re-invent the wheel to some extent. Unit tests have run-time dependencies, which are potentially substantially different to build-time dependencies. Ergo - to run 'build check' in a given module, we may want to check that a number of other things are built/delivered first - or (worse) all our tests may fail if they are not already there. We don't want to complicate & linearise the nice parallel build (even if there were no circular run-time dependencies - which there are ;-). Or am I confused ?
@mmeeks: But the proposed patch does not take care of any unit-test runtime dependencies, anyway!?
@kendy: I see your point about "check" switch/parameter... But I do not accept your arguments about "visibility of prj/tests.lst". In the description you wrote: "we introduced prj/tests.lst which has the same form as build.lst", so my proposal it's only about one -meaningful(!)- switch. If someone is able to make changes to tests.lst, making changes to build.lst will not be a problem. @mmeeks: perfect point! ;-). Thus, according to description ("building that module, and consequently running the tests"), the unified build.lst allows us better describe dependencies. Not only between test directories, but also between test dirs and "nmake" dirs... That is some test can be run when the module is not yet built completely. But still I think that makefile.mk solution is the best one
vg: Well, "If someone is able to make changes to tests.lst, making changes to build.lst will not be a problem." is very arguable ;-) I'm able to create a new line in prj/build.lst, or do changes there to fix parallel build issues, etc. but never knew what this 'nmake' means - the only explanation I saw was in http://wiki.services.openoffice.org/wiki/Hacking#prj.2Fbuild.lst [mmeeks, thanks for the article! :-)] Believe me, these files grow by copy and pasting; without the example of the separate file, we'll never see the amount of unit tests growing by the community programmers. sb: Yes, if we agree that we want to go the way I proposed, I'll happily extend that. But at the moment it does not make sense anyway - as you know, we have a circular dependency sal -> testshl2 -> sal, so... Please, let's talk in patches instead of in words; please show us your version, and we can compare - without it, it is hard to see the advantages and disadvantages...
kendy: as to complexity - any test developer must know which file should be changed and how (is it build.lst or test.lst - doesn't matter). He also should know how to write dependencies... So +one more field is not a problem in my opinion. Moreover, if we use makefile.mk with only target "check" - there's no even need to change 'nmake' field. About patches: I think we should first agree about strategy. What do we want to see at the end? Do you want to run tests after the project is built or (case of ring deps) - you might want to run them after the office is built? Both approaches are acceptable...
vg: Ideally both :-) - with my patches 'build check' in a module is possible, as well as 'build --all check' in instsetoo_native. From my point of view, both are necessary - the former for the developers changing something in a module, the latter eg. for us (OpenOffice.org, Novell, Sun, ...) for checks before release. wrt. your makefile.mk way, I still don't see what you mean by that, and how changing makefile.mk's _only_ would help us :-( - here an example patch would be really helpful.
> sb: But the proposed patch does not take care of any unit-test runtime > dependencies, anyway!? True not yet - OTOH, who said it was perfect ? should we wait until every patch is perfect to commit it ? if so, how did OO.o get into the repository ? ;-> > vg: perfect point! ;-) Sadly I didn't explain it clearly enough. > vg: the unified build.lst allows us better describe dependencies. Not only > between test directories, but also between test dirs and "nmake" dirs... That > is some test can be run when the module is not yet built completely. But the main dependency problems are not between project sub directories at all (the only scope of internal dependency that build has), but between projects: eg. ucbhelper's unit tests will require 'ucb' to be built, yet that is not currently a build dependency, nor should it need to be. ie. there is a -different- (run-time) dependency graph for unit-tests, than for building: and turning both graphs into the same thing is unnecessary and would be bad for build performance. Also - vg: in general it is anti-social to apply lots of stop-energy to something on which you have done no work. Something is better than nothing in the area of unit-tests IMHO - if it needs lots of re-writing in your own way later: that's fine; in the mean time - will you accept this ? Thanks.
mmeeks: "in general it is anti-social to apply lots of stop-energy to something on which you have done no work." You're funny guy, I admire your style of arguing ;-) "Something is better than nothing in the area of unit-tests IMHO - if it needs lots of re-writing in your own way" "Something" is not better then nothing when we can provide a proper implementation right from the beginning. World will not come down if we take time to think the problem over. later: that's fine; in the mean time - will you accept this? No, until I'm not convinced we do it the right way kendy: we discussed your solution here and decided that because of different dependency graph for tests, an additional tests.lst is the least of evils. But, in order to provide proper handling of global dependencies, it must have similar first line as build.lst has. So far the dependencies are broken, so it can initially be like: "pn prjname : NULL" So, with this approach we can: run only tests with say "build -a --onlytests" run regular build & tests in each built module with something like "build -a --withtests" The last scenario will merge all global build&test dependencies together, thus complicating the resulting dependency graph and eventually slowing down the build process... Of cause, if the dependency from tests.lst will cause ring dependencies in the resulting graph, the build will break... If everyone agrees with this proposal, we can start to work on patches...
> "Something" is not better then nothing when we can provide a proper > implementation right from the beginning. World will not come down > if we take time to think the problem over. True; the world will not end, but external contribution may cease, and disgruntled people may wander off and do something else, somewhere where they can be more effective for whatever reason :-) This is always a consideration in community building. It comes down to a question of the extent of the problem - flipping the question: is this something impossibly evily wrong that causes the world to end if integrated in it's current form ? Anyhow - thanks for considering the patch & commenting; personally I agree with your conclusion that we need an intra-module dependency list in the tests.lst, and that it is cheap & easy to base that from the build dependencies. > The last scenario will merge all global build&test dependencies together, thus > complicating the resulting dependency graph and eventually slowing down the > build process... Running the unit tests will even make the build slower too ;-) and if we defer running the complete set of unit tests until the build has ~completed we potentially save a certain amount of complexity; I expect a 'build check' inside a module to be the common developer use-case. HTH.
Why not just drop the test depemdencies. As I see it there are two usecases: 1. The developer wants to test her work. In this case a complete build is there already and everything needed is there also, so all dependencies are always satisfied. 2. a complete build should get tested. In this case why not have a clean build first and then start a testrun for each module. So again you don't need any dependencies. Just my 2c
adding myself to CC ...
I'm adding this comment to all open issues with Issue Type == PATCH. We have 220 such issues, many of them quite old. I apologize for that. We need your help in prioritizing which patches should be integrated into our next release, Apache OpenOffice 4.0. If you have submitted a patch and think it is applicable for AOO 4.0, please respond with a comment to let us know. On the other hand, if the patch is no longer relevant, please let us know that as well. If you have any general questions or want to discuss this further, please send a note to our dev mailing list: dev@openoffice.apache.org Thanks! -Rob