I'd like to take the same new data set and pass it to two versions of my text analysis model, trained on the same data, where the only difference is one model takes in an extra meta-data parameter (subject and body instead of just subject). Then compare the precision, recall and overall accuracy of both - similar to what I see if I were to just train & test a topic model.
Is there a tutorial how to do this today? I was taking the Decisioning Consultant courses but the Testing your simulation section didn't really cover this (plus doesn't cover 7.4, only 7.2).
Maybe there is, if you're talking about Adaptive Models: while the standard reporting from the Analytics Center (aka Prediction Studio in future releases) focusses on a single prediction (single rule), there is a section in the Report Browser that has a couple of out-of-the-box reports that can give a view across predictions.
From the CDH Portal you would launch the Report Browser from the top-right menu
From other applications that include the Analytics Center portal you would go to Reports on the left-hand side
Other applications might also give access to the standard Report Browser in other ways.
From there, choose the Adaptive Decisioning category
Then there's various reports to choose from. Their first screens often look similar, but there is difference in their drill-down capabilities.
For example, the "Adaptive Models Overview" report shows all Adaptive Models in the system. You can use the standard filtering options to filter.