Project:Data Dictionary/Proposals
From Step Mods | Change The Game
Mod Testing
- Form:SystemSpecs
- Template:SystemSpecs
- cpuspeed | CPUSpeed [Number]
- cpucore | CPUCore [Number(list)]
- gpumodel | GPUModel [String[list)]
- gpuvram | GPUVram [String(list)]
- multigpu | MultiGPU [Number(list)]
- ssd | SSD [Boolean]
- arch | OSArchitecture [Number(list)]
- hasdlc | HasDLC [String (list-multi select)]
- Template:SystemSpecs
- Form:ModTest
- Template:ModTest
- newGame | NewGame [Boolean]
- loadSave | CanLoadSave [Boolean]
- loadSaveName | LoadSaveName [String(list)]
- uninstall | CleanUninstall [Boolean]
- errors | HasErrors [Boolean]
- ctd | HasCTD [Boolean]
- bloat | SaveGameBloat [Boolean]
- Template:ModTest
- Form:Performance
- Template:Performance
- fps | FPSImpact [Number]
- vram | VRAMImpact [Number]
- Template:Performance
ModAttributes: Some of the attributes for this template should be associated with the Mod page itself.
Propose that we add to Template:ModTest:
- runningdlc | RunningDLC [String (list-multi select)]
RE Template:ModAttributes:
- @s4n, should the internal objects of the Documentation property be converted to property "string" values of that property? Could you re-create that one as you see fit?
- Should we add the QualityImpact property to the end of this?
We should rename Template:PerformanceImpact to Template:PerformanceData and use the previous name to hold the performance result (i.e., "impact")
RE Template:SystemSpecs & Template PerformanceData:
- How can we set these to evaluate to the SystemType and PerformanceImpact Properties, respectively? These can either evaluate directly in a 1:1 or in aggregate as a query result of the "average" of these respective templates ...
- * Yes to the Documentation properties. There should only be three (the ones currently designated as sub properties) all of type boolean.
- * Per the talk last night, SystemSpecs will be queried for the user entering PerformanceData and subsequently stored on the report page for that PerformanceData. Beyond this, we'll need a query that can pull results from the PerformanceData entered and process accordingly. Still not sure on how we're going to do that.
- Stoppingby4now (talk) 23:42, November 27, 2012 (UTC)
- *Agree with adding the Running DLC properties, also, I didn't add QualityImpact to either list as I wasn't sure if we were going to move it at the time. As I said on the forums, where the QualityImpact goes depends on whether we want to empirically collect that info per-tester or if we just want to leave that as one of the attribute items to be fleshed out in discussion.
- *What's the difference between PerformanceImpact and PerformanceData, or am I not understanding what you said? Doesn't all that info belong together?
- *It seems Semantic Result Forms provides quite a few ways of presenting data usefully, especially the charts and data visualizations sections.
- *In response to Stopping's forum post: don't we want PerformanceData and ModTest on the same report, thus the same form?
- ~Farlo
Talk 06:08, November 28, 2012 (UTC)
- * Re: Performance data. No, that should be a separate form. The main ModTest form will be tied to a single page(ticket) that represents the mod being tested. There should then be a link to the form for adding performance data. If you included it in the same form, management could become far too much to manage effectively depending on how often performance data is submitted.
- * Added list at the top as a starting point. Collapsed some things, expanded others. Discuss.
- * In the list above, collapsed the DLC's to a single property. No need to keep adding these when we can make it multi-select and not have to worry about updating the form/template every time a new DLC comes out.
- Stoppingby4now (talk) 06:17, November 28, 2012 (UTC)
- * PerformanceImpact = Low, Med, High (or just low/high) and PerformanceData holds specific FPS values. This is just an idea to aggregate results based on pre-defined commonalities or thresholds. Think of it like a frequency histogram (you have to create buckets for small-mid overall n sizes, else you just get a bunch of 1s for each FPS/VRAM combo). Otherwise, this may be accomplished by the query though, so I defer to Stopping on that (thinking about it further, this seems more useful, but I am not sure of the trade offs).
- * Agree with the idea of keeping Performance results separate from test results ... same goes for anything we are interested in collecting in aggregate.
- * Also agree with the new org above. I suggest that we use consistent field/property names though (unless we use some consistent differentiator (e.g., lower vs upper case).
- * I also like the idea of adding some if not all of the mod attributes to the mod pages (in the info box?)
- ~z929669
Talk 07:17, November 28, 2012 (UTC)
- * Alright, I'm a little confused. I thought each user was submitting their test results as far as whether the mod gives them CTDs, has errors, etc. individually but your explanation seems to imply that each mod/version only gets one ticket for this information. If that's the case, how are we going to determine the status of those properties between different machines with different results? If one tester does a test, doesn't get CTDs, and submits the report reflecting that, but then another tester does encounter CTDs, how are we going to handle that? I thought we were collecting this information per user/test and then compiling the results to see if we can find a correlation between certain system specs and certain errors. If we're leaving this kind of analysis/discussion/troubleshooting to a talk page/forum thread then that defeats a lot of the purpose in collecting the information semantically and it's not much better than bickering in the forums until we get enough yays/nays either way.
- * I like the changes to SystemSpecs, that should give us a lot more detailed info. I'm just curious, but how would we query based on DLC with the consolidated property; is it as easy as "if this property contains the sub-string "Dawnguard" somewhere in it"?
- * At least for the basic/attribute stuff, that'll definitely show up on the mod page, the question is whether we want to display some kind of generalization of the test/performance results as well. If we do, then that involves creating some rough guidelines so those aren't declared entirely subjectively.
- ~FarloTalk 07:38, November 28, 2012 (UTC)
- * Re: template vs form names. I'm coming at this from the perspective of working with all of this in source, because the Semantic tools (creating forms, templates, etc.) are very limited in what they can do, and ultimately you are going to have to get your hands dirty (I create everything from scratch in source). When names are exactly the same, it's far more annoying to work with especially when you are editing after the fact. Refer to Template:Mod. I had changed some of them for this very reason because it was just too annoying having them exactly the same (ex. fps -> AffectsFPS, forumtid -> ForumTID, core -> IsCore, etc.). They still represent IMO, and in some cases are far easier to type (template fields also have the potential to be used many times, where-as properties are mainly used only to assign a value). But I'm also a heavy scripter (think programming where you pass in a value to a function/method, you don't want the parameter to be the same name as an existing internal variable). The absolute minimum I would be willing to accept is beginning lower-case for template fields, and upper-case for properties, but I will ask you to at least consider the above.
- * Farlo makes a good point. We still want a single "ticket" per mod as a collection point, but perhaps that should just be a very simple form purely for creating the page, and maybe providing more basic information. ModTest could then be another entry point for users to submit information, and results can be automatically updated on the mod ticket page based on that information. For example, if just one user checks that they experienced a CTD, the mod would then be marked as having had a CTD so it can be examined further. We would also need a comment field for users to enter more detailed information. But we would still need a mechanism for overriding such input for cases such as false positives.
- * Re: DLC. Yes, you can query on a property to see if it contains a specific value when it contains a list of items.
- Example: {{#ask: [[DLC::Dawnguard]] ... }}. You can think of it as an array, but you don't have to specify an index. Just search for the data you are looking for, and if it's there the condition is true.
- Stoppingby4now (talk) 09:56, November 28, 2012 (UTC)
- * RE upcase vs lowcase fields/properties. Go ahead and make the call, but I think that all fields should be lower case and all properties should be proper or upper case for visual purposes. However, it is often difficult and nonsensical to think up different NAMES for essentially the same thing (which is more what I think we are dealing with in the SMW "scripting" case). I hate trying to come up with a unique identifier for the same concept for no more reason than to differentiate between two different types of metadata.
- * RE the Performance Data and Mod-Test Data: Any Property that is variable could be treated like the Performance Data (i.e., meaningfully aggregated). HasErrors is definitely not variable and several other Properties might not be either (e.g., NewGame, CleanUninstal, LoadNewGame, SaveGameBloat); however, these latter could vary depending on the DLC background or even the Windows environment. Therefore, it does make sense to geoup these with Performance Data ... BUT, I predict that the variation will be low or nil for most. Nevertheless, it might be wise to group in Performance Data ... need to see some real examples of possible reports to be sure (Semantic Result Formats does sound intriguing here).
- * Updated the breakout with respect to DLC Properties (changed name in SystemSpecs and added similar to ModTest)
- ~z929669Talk 17:01, November 28, 2012 (UTC)
- I agree that most of the basic information only needs to be 1:1 with the mod, such as documentation and conflicts, but I'm thinking in terms of users having different errors and setups when they submit this kind of information. Yes, it technically isn't performance related (unless crashing means its performance is null), but say we have 20 testers/systems test a mod and even 4 of them get CTDs (or any other errors/crashes for that matter), that doesn't bode well when the sample size is in the thousands. If the CTD, Papyrus, and Savegame properties are 1:1 with a mod then it basically becomes a forum discussion trying to figure out what's going on and the semantic data is meaningless. On top of that, it means the testers need to go through another step to report the CTD by going to the appropriate thread/talk page and posting manually. This means we no longer have a semantic count of how many people are getting errors and all analysis of the problem is done manually. If we collect each tester's experience with these errors along with the PerformanceData on the same report then we can see the "spread" of errors and maybe even draw some preliminary conclusions based on that test's SystemSpecs or PerformanceData. I'd rather collect this information individually, post the aggregate on the talk page, and discuss from there, at least then we have some empirical numbers to run with.
- I also think it'd be a good idea to give the testers an open comment box next to some of the fields, such as CTDs, where they can elaborate on problems. Obviously this isn't very useful semantically but with a simple query we can get a page of everyone's experience/problems, and that's a lot easier to parse through than a talk page where everyone's talking and troubleshooting at once. As I said above, it also means testers don't have to go to another page to elaborate on their errors/crashes and (if the tester explains properly) at a glance it gives us an idea of where they are when these problems happen and can help determine which reports have false positives without having to rely on the tester posting on a talk page.
- ~FarloTalk 19:11, November 28, 2012 (UTC)
- * RE: template fields/properties. I agree with making template fields lowercase and Semantic Properties CamelCase. In terms of having them different, take a look at my examples again. I'm by no means coming up with random names just for the sake of making them different. Everything has an association still based on the premise it is meant to convey. fps->AffectsFPS, bloat->SaveGameBloat, etc. When you start to get long names, lowercase quickly loses its meaning. And again, properties are used once in a template, whereas template fields can be re-used. For the two examples above, I have no problem knowing what those map to, and it makes typing them multiple times much easier (templates are localized, so you always know what you are working with based on the template).
- * In terms of 1:1 relation of information to a Mod, I believe it still needs to be there. Even if you only get 4 CTD's out of a 1000, that by itself shouldn't disqualify the potential that there is still a problem. It may just be that the problem is very rare, and those 4 happen to have the right conditions for it to occur. Semantics aren't lost in this case either, as you want to collect this information to know what users are experiencing. This is why I also made the comment that we would need some means of overriding it at the Mod level. It may be that those 4 CTD's/1000 are not valid,and we choose to continue to use it based on that information. But what if they are valid? But, there is still no avoiding the need for open discussion for cases such as this to attempt to find problems.
- * Concerning textboxes for detailed input, I also agree. The big question though is do we really need a comment box per boolean? I would tend to lean towards having a single comment box per "report" that is being submitted, as separating them can make locating data far more cumbersome. I can see the case for pulling comments on CTD's for example, but then you have multiple queries for each individual boolean attribute, and when focusing so narrowly, you lose site of the overall context of the report that was submitted.
- Stoppingby4now (talk) 23:50, November 28, 2012 (UTC)
- * I agree that at the end of the day, we need to manually discuss/agree on a specific mod being "CTD free" (or equivalent) and set a Boolean for that, but from what I gathered of your posts you didn't wan to collect the data from users aside from in comment form. I'd suggest we have users submit the information along with PerformanceData individually, and then we discuss said results and manually reach a conclusion on whether the mod as a whole "passes" or not. Obviously this means we'd need another property and that introduces the problem of keeping report-specific properties separate from mod-specific properties, but IMO this is worth doing as it gives a way of semantically searching for "approved" or CTD-causing mods at one level while allowing us to investigate an individual mod based on its reports.
- * I also agree that just having one comment box is best, as long as they aren't typing out multiple paragraphs.
- ~FarloTalk 00:10, November 29, 2012 (UTC)
- * Actually, I had agreed with you in regard to users being able submit the data separately several posts back. :P My mention of comments was purely in relation to user discussion as is the case with all bug tracking implementations, and will be needed in some form without requiring an official report being submitted. In other words, comments tied to a report such as PerformanceData should be all about providing the detail necessary for the problem, but we still need a means of allowing discussions without requiring additional reports being submitted. Doing so would break direct association and make things much harder to track.
- * This seems to be moving (and I think correctly) in the direction of just having a mod page to act as the glue for reports based on that mod. Each mod would then have bug reports submitted in association with it, and we need the ability for users to discuss on each submitted report. But, those reports still need to affect the mod "state" during the development/testing phase. I picture it more in terms of users submitting reports which allows for others to corroborate the results. We (STEP Team) then set the final state of the Mod based on the above results. This is still very important, as we will need this information between testing phases.
- * However, with all of that said, there are still two distinct sets of data that we need to deal with. On the one hand, we have performance data (fps, vram, etc.) that can be submitted (n) times per user, and is useful for getting better overall overages. But, in relation to a specific problem, there should only be one "ticket". Allowing multiple reports for the identical problems is not realistic and makes things far too difficult to track. I'm beginning to think that the report needs to have a dropdown list for the user to select the type of problem they are reporting about (CTD, SaveGameBloat, etc.), so that problems can be tracked specifically. Trying to cram tons of reports, all having the potential to contain information on more than one problem is going to be a nightmare.
- Stoppingby4now (talk) 01:32, November 29, 2012 (UTC)
- Created [[Template:Ri]] for the text above, we've indented waaaaay too much. EDIT: It's a useless template and was deleted. Just don't use the colons, duh. ~z929669
Talk 19:31, January 8, 2021 (UTC) - Ah, well then I could have stopped babbling a while ago :P There's definitely going to be discussion about the results and how we're going to declare the mod properties, and I think it might be a good idea to make a "/ReportTalk" page for each mod to centralize all the discussion. The dedicated page will also allow us to (hopefully automatically through templates) post some basic aggregations to start the ball rolling, or at least give a basic overview of the results that have been submitted.
- With regards to the mod's page gluing everything together, I completely agree, and that's why I originally proposed we store the reports as sub-pages of the mod itself, although I see now that'd be a hassle telling the form where to create the page (or would it?). In either case tying everything to the mod page, at least semantically, is a good idea. Then each mod will eventually have a series of Boolean (maybe three options for some?) that mark a mod as having passed testing. As far as commenting on individual reports there's always the option of using the report's talk page, but that can break down if people aren't familiar with Wikiquette or how the pages are laid out, but then again teaching testers the basics of the markup and how to have threaded conversations here will likely be required.
- I wish making mock-ups was easier, but from the sound of it I like your idea. If only we had a whiteboard, then we could draw bubble maps, big arrows, and important sounding keywords to show connections. Some visualization might help us grasp this whole process a lot better. I wonder if there's anything like Teamviewer's drawing tools that are web-based. I guess we could use a JPG in Dropbox, but that's a little primitive.
