Project:Data Dictionary/Proposals

From Step Mods | Change The Game
Revision as of 00:10, November 29, 2012 by Farlo (talk | contribs)

Mod Testing

  • Form:SystemSpecs
    • Template:SystemSpecs
      • cpuspeed | CPUSpeed [Number]
      • cpucore | CPUCore [Number(list)]
      • gpumodel | GPUModel [String[list)]
      • gpuvram | GPUVram [String(list)]
      • multigpu | MultiGPU [Number(list)]
      • ssd | SSD [Boolean]
      • arch | OSArchitecture [Number(list)]
      • hasdlc | HasDLC [String (list-multi select)]
  • Form:ModTest
    • Template:ModTest
      • newGame | NewGame [Boolean]
      • loadSave | CanLoadSave [Boolean]
      • loadSaveName | LoadSaveName [String(list)]
      • uninstall | CleanUninstall [Boolean]
      • errors | HasErrors [Boolean]
      • ctd | HasCTD [Boolean]
      • bloat | SaveGameBloat [Boolean]
  • Form:Performance
    • Template:Performance
      • fps | FPSImpact [Number]
      • vram | VRAMImpact [Number]

ModAttributes: Some of the attributes for this template should be associated with the Mod page itself.


Propose that we add to Template:ModTest:

  • runningdlc | RunningDLC [String (list-multi select)]

RE Template:ModAttributes:

  • @s4n, should the internal objects of the Documentation property be converted to property "string" values of that property? Could you re-create that one as you see fit?
  • Should we add the QualityImpact property to the end of this?

We should rename Template:PerformanceImpact to Template:PerformanceData and use the previous name to hold the performance result (i.e., "impact")

RE Template:SystemSpecs & Template PerformanceData:

  • How can we set these to evaluate to the SystemType and PerformanceImpact Properties, respectively? These can either evaluate directly in a 1:1 or in aggregate as a query result of the "average" of these respective templates ...
~z929669Ixian Insignia.pngTalk 21:00, November 27, 2012 (UTC)
* Yes to the Documentation properties. There should only be three (the ones currently designated as sub properties) all of type boolean.
* Per the talk last night, SystemSpecs will be queried for the user entering PerformanceData and subsequently stored on the report page for that PerformanceData. Beyond this, we'll need a query that can pull results from the PerformanceData entered and process accordingly. Still not sure on how we're going to do that.
Stoppingby4now (talk) 23:42, November 27, 2012 (UTC)
*Agree with adding the Running DLC properties, also, I didn't add QualityImpact to either list as I wasn't sure if we were going to move it at the time. As I said on the forums, where the QualityImpact goes depends on whether we want to empirically collect that info per-tester or if we just want to leave that as one of the attribute items to be fleshed out in discussion.
*What's the difference between PerformanceImpact and PerformanceData, or am I not understanding what you said? Doesn't all that info belong together?
*It seems Semantic Result Forms provides quite a few ways of presenting data usefully, especially the charts and data visualizations sections.
*In response to Stopping's forum post: don't we want PerformanceData and ModTest on the same report, thus the same form?
~FarloUser Farlo Sig.pngTalk 06:08, November 28, 2012 (UTC)
* Re: Performance data. No, that should be a separate form. The main ModTest form will be tied to a single page(ticket) that represents the mod being tested. There should then be a link to the form for adding performance data. If you included it in the same form, management could become far too much to manage effectively depending on how often performance data is submitted.
* Added list at the top as a starting point. Collapsed some things, expanded others. Discuss.
* In the list above, collapsed the DLC's to a single property. No need to keep adding these when we can make it multi-select and not have to worry about updating the form/template every time a new DLC comes out.
Stoppingby4now (talk) 06:17, November 28, 2012 (UTC)
* PerformanceImpact = Low, Med, High (or just low/high) and PerformanceData holds specific FPS values. This is just an idea to aggregate results based on pre-defined commonalities or thresholds. Think of it like a frequency histogram (you have to create buckets for small-mid overall n sizes, else you just get a bunch of 1s for each FPS/VRAM combo). Otherwise, this may be accomplished by the query though, so I defer to Stopping on that (thinking about it further, this seems more useful, but I am not sure of the trade offs).
* Agree with the idea of keeping Performance results separate from test results ... same goes for anything we are interested in collecting in aggregate.
* Also agree with the new org above. I suggest that we use consistent field/property names though (unless we use some consistent differentiator (e.g., lower vs upper case).
* I also like the idea of adding some if not all of the mod attributes to the mod pages (in the info box?)
~z929669Ixian Insignia.pngTalk 07:17, November 28, 2012 (UTC)
* Alright, I'm a little confused. I thought each user was submitting their test results as far as whether the mod gives them CTDs, has errors, etc. individually but your explanation seems to imply that each mod/version only gets one ticket for this information. If that's the case, how are we going to determine the status of those properties between different machines with different results? If one tester does a test, doesn't get CTDs, and submits the report reflecting that, but then another tester does encounter CTDs, how are we going to handle that? I thought we were collecting this information per user/test and then compiling the results to see if we can find a correlation between certain system specs and certain errors. If we're leaving this kind of analysis/discussion/troubleshooting to a talk page/forum thread then that defeats a lot of the purpose in collecting the information semantically and it's not much better than bickering in the forums until we get enough yays/nays either way.
* I like the changes to SystemSpecs, that should give us a lot more detailed info. I'm just curious, but how would we query based on DLC with the consolidated property; is it as easy as "if this property contains the sub-string "Dawnguard" somewhere in it"?
* At least for the basic/attribute stuff, that'll definitely show up on the mod page, the question is whether we want to display some kind of generalization of the test/performance results as well. If we do, then that involves creating some rough guidelines so those aren't declared entirely subjectively.
~FarloUser Farlo Sig.pngTalk 07:38, November 28, 2012 (UTC)
* Re: template vs form names. I'm coming at this from the perspective of working with all of this in source, because the Semantic tools (creating forms, templates, etc.) are very limited in what they can do, and ultimately you are going to have to get your hands dirty (I create everything from scratch in source). When names are exactly the same, it's far more annoying to work with especially when you are editing after the fact. Refer to Template:Mod. I had changed some of them for this very reason because it was just too annoying having them exactly the same (ex. fps -> AffectsFPS, forumtid -> ForumTID, core -> IsCore, etc.). They still represent IMO, and in some cases are far easier to type (template fields also have the potential to be used many times, where-as properties are mainly used only to assign a value). But I'm also a heavy scripter (think programming where you pass in a value to a function/method, you don't want the parameter to be the same name as an existing internal variable). The absolute minimum I would be willing to accept is beginning lower-case for template fields, and upper-case for properties, but I will ask you to at least consider the above.
* Farlo makes a good point. We still want a single "ticket" per mod as a collection point, but perhaps that should just be a very simple form purely for creating the page, and maybe providing more basic information. ModTest could then be another entry point for users to submit information, and results can be automatically updated on the mod ticket page based on that information. For example, if just one user checks that they experienced a CTD, the mod would then be marked as having had a CTD so it can be examined further. We would also need a comment field for users to enter more detailed information. But we would still need a mechanism for overriding such input for cases such as false positives.
* Re: DLC. Yes, you can query on a property to see if it contains a specific value when it contains a list of items.
Example: {{#ask: [[DLC::Dawnguard]] ... }}. You can think of it as an array, but you don't have to specify an index. Just search for the data you are looking for, and if it's there the condition is true.
Stoppingby4now (talk) 09:56, November 28, 2012 (UTC)
* RE upcase vs lowcase fields/properties. Go ahead and make the call, but I think that all fields should be lower case and all properties should be proper or upper case for visual purposes. However, it is often difficult and nonsensical to think up different NAMES for essentially the same thing (which is more what I think we are dealing with in the SMW "scripting" case). I hate trying to come up with a unique identifier for the same concept for no more reason than to differentiate between two different types of metadata.
* RE the Performance Data and Mod-Test Data: Any Property that is variable could be treated like the Performance Data (i.e., meaningfully aggregated). HasErrors is definitely not variable and several other Properties might not be either (e.g., NewGame, CleanUninstal, LoadNewGame, SaveGameBloat); however, these latter could vary depending on the DLC background or even the Windows environment. Therefore, it does make sense to geoup these with Performance Data ... BUT, I predict that the variation will be low or nil for most. Nevertheless, it might be wise to group in Performance Data ... need to see some real examples of possible reports to be sure (Semantic Result Formats does sound intriguing here).
* Updated the breakout with respect to DLC Properties (changed name in SystemSpecs and added similar to ModTest)
~z929669Ixian Insignia.pngTalk 17:01, November 28, 2012 (UTC)
I agree that most of the basic information only needs to be 1:1 with the mod, such as documentation and conflicts, but I'm thinking in terms of users having different errors and setups when they submit this kind of information. Yes, it technically isn't performance related (unless crashing means its performance is null), but say we have 20 testers/systems test a mod and even 4 of them get CTDs (or any other errors/crashes for that matter), that doesn't bode well when the sample size is in the thousands. If the CTD, Papyrus, and Savegame properties are 1:1 with a mod then it basically becomes a forum discussion trying to figure out what's going on and the semantic data is meaningless. On top of that, it means the testers need to go through another step to report the CTD by going to the appropriate thread/talk page and posting manually. This means we no longer have a semantic count of how many people are getting errors and all analysis of the problem is done manually. If we collect each tester's experience with these errors along with the PerformanceData on the same report then we can see the "spread" of errors and maybe even draw some preliminary conclusions based on that test's SystemSpecs or PerformanceData. I'd rather collect this information individually, post the aggregate on the talk page, and discuss from there, at least then we have some empirical numbers to run with.
I also think it'd be a good idea to give the testers an open comment box next to some of the fields, such as CTDs, where they can elaborate on problems. Obviously this isn't very useful semantically but with a simple query we can get a page of everyone's experience/problems, and that's a lot easier to parse through than a talk page where everyone's talking and troubleshooting at once. As I said above, it also means testers don't have to go to another page to elaborate on their errors/crashes and (if the tester explains properly) at a glance it gives us an idea of where they are when these problems happen and can help determine which reports have false positives without having to rely on the tester posting on a talk page.
~FarloUser Farlo Sig.pngTalk 19:11, November 28, 2012 (UTC)
* Good points, and I agree on adding open-ended comment boxes next to Papyrus and CTD (possibly other non-Boolean).
~z929669Ixian Insignia.pngTalk 20:44, November 28, 2012 (UTC)
* RE: template fields/properties. I agree with making template fields lowercase and Semantic Properties CamelCase. In terms of having them different, take a look at my examples again. I'm by no means coming up with random names just for the sake of making them different. Everything has an association still based on the premise it is meant to convey. fps->AffectsFPS, bloat->SaveGameBloat, etc. When you start to get long names, lowercase quickly loses its meaning. And again, properties are used once in a template, whereas template fields can be re-used. For the two examples above, I have no problem knowing what those map to, and it makes typing them multiple times much easier (templates are localized, so you always know what you are working with based on the template).
* In terms of 1:1 relation of information to a Mod, I believe it still needs to be there. Even if you only get 4 CTD's out of a 1000, that by itself shouldn't disqualify the potential that there is still a problem. It may just be that the problem is very rare, and those 4 happen to have the right conditions for it to occur. Semantics aren't lost in this case either, as you want to collect this information to know what users are experiencing. This is why I also made the comment that we would need some means of overriding it at the Mod level. It may be that those 4 CTD's/1000 are not valid,and we choose to continue to use it based on that information. But what if they are valid? But, there is still no avoiding the need for open discussion for cases such as this to attempt to find problems.
* Concerning textboxes for detailed input, I also agree. The big question though is do we really need a comment box per boolean? I would tend to lean towards having a single comment box per "report" that is being submitted, as separating them can make locating data far more cumbersome. I can see the case for pulling comments on CTD's for example, but then you have multiple queries for each individual boolean attribute, and when focusing so narrowly, you lose site of the overall context of the report that was submitted.
Stoppingby4now (talk) 23:50, November 28, 2012 (UTC)
* I agree that at the end of the day, we need to manually discuss/agree on a specific mod being "CTD free" (or equivalent) and set a Boolean for that, but from what I gathered of your posts you didn't wan to collect the data from users aside from in comment form. I'd suggest we have users submit the information along with PerformanceData individually, and then we discuss said results and manually reach a conclusion on whether the mod as a whole "passes" or not. Obviously this means we'd need another property and that introduces the problem of keeping report-specific properties separate from mod-specific properties, but IMO this is worth doing as it gives a way of semantically searching for "approved" or CTD-causing mods at one level while allowing us to investigate an individual mod based on its reports.
* I also agree that just having one comment box is best, as long as they aren't typing out multiple paragraphs.
~FarloUser Farlo Sig.pngTalk 00:10, November 29, 2012 (UTC)