Cranes at CVR Q23

NLP Analysis on Frankenstein (Full Book Text)

Dataset:

Use the file frankenstein.txt containing the full text of Mary Shelley's novel. frankenstein.txt

---

Part A – Preprocessing and Basic Statistics

1. Read the full text from frankenstein.txt.
2. Remove Gutenberg headers and footers (if present).
3. Convert text to lowercase and remove punctuation.
4. Tokenize into words and compute:
   • Total number of words
   • Number of unique words
   • Average word length
   • Total number of sentences
   • Average sentence length (in words)

---

Part B – Vocabulary and Frequency

5. Find and display the top 20 most frequent words (after removing stopwords).
6. List 15 words that appear only once (hapax legomena).
7. Extract and display the 10 longest words in the text.

---

Part C – Named Entity and Character Analysis

8. Use POS tagging or NER to extract proper nouns (names, places).
9. Count how many times each major character appears (e.g., Victor, Frankenstein, monster, Elizabeth).
10. Find 5 sentences where both "Victor" and "monster" appear together.

---

Part D – Thematic and Chapter Analysis

11. Split the novel by chapter (using the keyword "Chapter").
12. For each chapter:
    • Count the number of words
    • Find top 5 words (excluding stopwords)
13. Track emotion using a lexicon (e.g., NRC or TextBlob):
    • Compute emotion/sentiment scores per chapter
    • Plot the sentiment progression across chapters
14. Implement KWIC (Keyword-in-Context) for terms like:
    • “life”, “death”, “science”, “creation”

---

Part E – Stylometric and Dialogue Analysis

15. Compute the type-token ratio (unique words / total words).
16. Compare the ratio across early and later chapters.
17. Detect passive voice constructions (e.g., “was created”, “had been abandoned”).
18. Extract all direct speech (lines within quotation marks).
19. Compare sentiment of spoken dialogue vs. narration.

---

Part F – Advanced NLP (Optional)

20. Apply topic modeling (LDA) to extract 5 major topics from the text.
21. Use TF-IDF and cosine similarity to compare Chapter 1 with Chapter 10.
22. Generate a summary of the entire book using:
    • Extractive method (TextRank)
    • Abstractive method (Transformer model)
23. Build a binary classifier to distinguish narration vs. dialogue using linguistic features.
24. Identify most common POS sequences and compare between narration and dialogue.

---

Output Requirements

• Print statistics and sample outputs clearly.
• Use visualizations for frequencies, emotions, and comparisons.
• Include sample quotes or snippets in contextual tasks.