Hello,

We are looking at using XStudio for managing test campaigns and running manual and automated tests. It would be great to hear the opinions of real users. There are a few interesting posts in the forums already, like these:
continuity-of-xqual-t218.html
current-project-approach-in-xqual-t281.html
xstudio-as-a-replacement-for-test-director-t267.html

Specifically, it would be really helpful I think if someone could answer these:

* I wonder how would XStudio perform with many users, say 50, using it and running test campaigns? I see that there are not different projects in XStudio but I think you could run different XStudio servers for different projects.

* I wonder how XStudio would handle test cases which take a VERY long time to complete? We would like to test a variety of systems and from almost unit-tests to system and performance tests.

* How robust and configurable for different types of tests are the XAgents? I’m going to try hooking up the XAgent to run tests based on tcl and also other languages/systems and evaluate the test running behaviour. This should give me a basic idea of the XAgent capabilities

* From looking at the history of the tool and looking on the forums I don’t think it has very widespread use so it might not be very “mature”. But a good few people on the forums use it and like it – which is nice! The interface to XQual XStudio seems very nice and feature-full. The free nature is fantastic but a big concern is the closed-source nature ( I know this has been discussed before and I understand why it is that way).

A bit of background for us: We’re already using Salome_TMF here. Salomé is a very “mature” framework (but also somewhat of a mess in ways inside)! Salomé is open source so we can fix it and add to it and make it do what we want … but it is weak on running the tests. It doesn’t pass off running of tests to an agent so there are various usage problems associated with that.

Thank you and best regards,

James.

Hi James,
As the developer of XStudio I cannot give a objective opinion but I can at least give some technical answers to some of the points you mentionned:



I made some big effort lately to improve performance and scalability. I also worked a lot on tracking potential memory leaks. I used to find some in the past (due to external libs leakages) but I think they are all fixed now...
About the projects, everything is organized in trees/folders so it is very likely that you can have all your projects on a single server db instance...



Again, I improved a lot performance and stability recently so it shouldn't be a problem but of course during all the testing time the Database must be accessible by XStudio/Xagent.



Personally, I used the agents for manual testing and installation testing without any major problem. Each type of test will have a specific launcher and configurations. I'm using currently ~10 different launchers concurrently without any problem.



You're right, XStudio is still young (2 years old) and hence not yet very mature but my stats show that the ppularity/usage of XStudio is growing very fast at the moment. The number of users doubled in the last 5 month.



Never used Salome but I have the feeling this is currently the more installed free test management system and it is very mature... however, I think XStudio has a better architecture (or I would have personnally used Salome initially and would never have started this project )

Hope this helps,

Hello Eric(I think ),

Yes your reply does help a good bit!
Thank you very much.
Some things you mention give a good insight into behaviour of the tool.
I haven't tested the tool very much from the usage side but does feel nice and fast to get around and work with.

I was looking at running Tcl tests and it was easy to modify the perl launcher to be a tcl launcher. It wasn't mentioned anywhere I saw that source code for the launchers was included with XStudio and XAgent, but thanks for that!
A developers guide is referred to in some places but perhaps it means just the javadocs?

Thanks!

Anyone have any non biased opinions? ...

James.

yes, this is Eric

I should probably make it clearer on the website that the sources of all the launchers (except the manual launchers - for technical reasons) are included (currently I guess only the installer shows the option to install it or not).

The javadoc is provided but the developer's guide is not ready yet. I must find some time to work on it...

Anyway, I'll be happy to help you setting up your environment if you have any issue.
Thx,

PS: Note that I'm also offering (only for a very few customers so far) some professional support (development of launchers, yearly support, customization, etc.) in case you would be interested in the future.

Hello and thanks Eric

Yes, I think it's a good idea to say the launcher source is included and if you showed the launcher source directly on the website it might be good. Do you mind if the tcl one is posted in a blog somewhere?

The install and set-up of XStudio is very easy at least so I'm definately don't need help setting up. Thanks for the offer of support It is good to know you are open to it if we need.

I found import of batches of test plans okay but limited to a certain extent. I used the .csv import, so fields like test description and other details had to be filled in manually or by writing them into database directly. It's worth nothing that the whitespace (indentation and seperation lines) are used in the parsing of the .csv. I wan not sure about generation of all the fields for the .xml format.

In general some more export/import interfaces to XStudio would be valuable, ...
e.g. for launcher ability to pass more details on test to script and attach extra message to test result (e.g. message visible in test reasults, link to test log area) Makes running and working with tests easier with XStudio.

For now I have to get some real work done and stop focussing on evaluating XStudio! (for now!)

James.

thanks,

Hi James,
I found by chance your blog where you explain how you implemented the Tcl launcher starting from the Perl one, I can see it was pretty easy ! Always good to see that the interface provided is clear/easy enough.

I plan to create an open-source section on XQual website where I will put the source of all the launchers. Can I include yours (the Tcl one) in this section and include it in the future releases of XStudio?

Thanks,