To find the top n most frequent words in a list of sentences, you can follow these steps:
-
Tokenize and Count: Iterate through each sentence, split it into individual words (tokens), and use a hash map (like std::unordered_map in C++) to store the frequency of each word. The word will be the key, and its count will be the value.
-
Prioritize and Sort: After counting all words, you need to identify the top n. A common approach is to use a min-heap (like std::priority_queue in C++ configured as a min-heap) of size n. Iterate through your word-frequency map. For each word-frequency pair:
- If the heap has fewer than n elements, add the pair.
- If the heap is full (has n elements) and the current word's frequency is greater than the frequency of the word at the top of the heap (the minimum frequency in the heap), remove the top element and insert the current word-frequency pair.
- If frequencies are equal, prioritize lexicographically smaller words (if required by the problem statement).
-
Extract Results: Once you've processed all word-frequency pairs, the min-heap will contain the n most frequent words. Extract them from the heap and reverse the order if necessary to get the final result (e.g., from most frequent to least frequent).
This method efficiently handles large datasets by maintaining a limited-size heap, ensuring a time complexity typically around O(N log k), where N is the total number of words and k is the number of top frequent words to find.