Seven principles of inductive software engineering

Seven principles of inductive software engineering

1. #1 Humans Before Algorithms

   • humans need to
     • understand the results
     • understand that those results add value to their work.
   • talking in “the right way”
     • Do they bring their senior management to the meetings?
     • Do they keep interrupting (you or each other) and debating your results?
     • Do they indicate they understand your explanation of the results?
     • Do your results touch on issues that concern them?
     • Do they offer more data sources for analysis?
     • Do they invite you to their workspace and ask you to teach them how to do XYZ?

2. #2 Plan for Scale

   • methods are usually repeated multiple times
     • answer new questions
     • enhance data mining method or fix some bugs
     • deploy the results
   • Another issues is that as you scale up, your methods will need to scale up as well.

3. #3 Get Early Feedback

   • try applying very simple tools to gain rapid early feedback.
   • simplicity first!
   • Get feedback early and often!

4. #4 Be Open Minded

   • mantras to chant while data mining
     • Avoid a fixed hypothesis
     • Avoid a fixed approach for data mining
     • seek important results.

5. #5 Be Smart with Your Learning

   • check and validate your results:
     • Ask other people to do code reviews of your scripts.
     • Check the conclusion stability against different sample policies.
     • don’t just report mean results

6. #6 Live with the Data You Have

   • spend some time on data quality operators
     • Use feature selection to remove spurious attributes.
     • Use row selection to remove outliers and group-related rows into sets of clusters.

7. #7 Develop a Broad Skill Set That Uses a Big Toolkit

   • important since many problems need specialized methods to find good solutions.
   • look for the “big ecology” toolkits where lots of developers are constantly trying out new ideas.