The manner in which numbers are disseminated in a government dataset poses a headache for data journalists. A majority of such datasets are typically in PDF format. Many are essentially image files converted into PDFs using online tools. This means that it is routine for a data journalist in India to spend 80%-90% of their time transforming PDF files into Excel sheets. The tedious task involves “cleaning” up the files by eliminating merged rows and columns to ensure that unique rows are ready for analysis and removing duplicates and blank rows. To do this. data journalists use tools such as Tabula and Adobe’s PDF to Excel converter. However, these are time-consuming efforts and often fail to deliver accurate results.
Data journalists, especially those who work in newsrooms, are constrained by deadlines. This means that they have limited time to analyse and draw conclusions from such data. OpenAI’s code interpreter has considerably eased this process. The Code Interpreter converts PDF documents into Comma Separated Files (CSV) and allows the journalist to download them for further analysis. Moreover, if the CSV files had merged rows and columns, a big no-no for data analysis, Code Interpreter makes them unique.
To leverage these capabilities, it’s crucial that the prompts are correctly formed. The hyperlinked Twitter thread provides a series of effective prompts, which assisted The Hindu’s data team in successfully cleansing an RTI response from the Department of Higher Education. The following is the story shaped by harnessing the data-cleaning capabilities of the Code Interpreter.
