Preprocessing

Preprocessing

1. wget corpus.zip

   • unzip, move all PDF files into all_txt folder, remove not needed folders
     • convert PDF to TEXT files
       • wget && unzip causeminer =PDF optimizer to remove headers & footers
         • clean_txt_files()
           • glob = load fulltext into a list
             • Create a DataFrame with full-text & titles
               • replace="dataset/all_optimized_txt/" with value=""
                 • df_test: remove all rows with len(Filename) < 30 ex. icis2015_1443.txt
                   • temp_df: split filename into columns "Authors", "Year", "Title", "Journal", "Version"
                     • fulltext_df: pd.concat([df_test, temp_df], axis=1)
                       • manually resolving duplicates in fulltext - remove rows by index
         • clean_txt_files() with Regexp <cite> replacement

2. labeled corpus: pd.read_excel

   • Remove rows that have DQ issues = ambiguous titles Title = "Editorial", "Editorial Notes", "Editorial Introduction"
     • remove Round column, then merge every 2nd row together through grouping by non-label columns and summing-up label-columns (as those are 0,1 values)
       • FINALLY: merge (inner join) fulltext df & labeled corpus by Title, result 129 rows only!
         • Question: should I manually change the file names ex. icis_1224.txt to correct form Authors_Title_Year_Version? = this way I can take them into account
           • should I manually research the correct title names for "Editorial", "Editorial Notes" etc?