Tokenize a text into words

Use the BreakIterator class. First get an instance through the factory method getWordInstance, initialize it with the text using setText and iterate through the elements with first and next.

This example uses the following text: text.txt:

It is, in fact, nothing short of a miracle that the modern methods 
of instruction have not yet entirely strangled the holy curiosity 
of inquiry; for this delicate little plant, aside from stimulation, 
stands mainly in need of freedom; without this it goes to wrack and 
ruin without fail. It is a very grave mistake to think that the 
enjoyment of seeing and searching can promoted by means of coercion 
and sense of duty. To the contrary, I believe that it would be 
possible to rob even a healthy beast of prey of its voraciousness, 
if it were possible, with the aid of a whip, to force the beast to 
devour continuously, even when not hungry,especiallyif the food, 
handed out under such coercion, were to be selected accordingly. 

Albert Einstein 

Main.java:

import java.text.*;
import java.io.*;
  
public class Main {
   public static void main(String args[]) {
      if (args.length != 1) {
         System.err.println("Usage: java Main <file.txt>");
         System.exit(1);
      }
 
      String text = readFile(args[0]);
 
      BreakIterator biWord = BreakIterator.getWordInstance();
      biWord.setText(text);
      System.out.println("Splitting text up into wordsn");
      printInstances(biWord, text);
   }  
 
   public static void printInstances(BreakIterator bi, String text) {
      int start = bi.first();
      int end = bi.next();
      while (end != BreakIterator.DONE) {
         System.out.println("Found: " + text.substring(start, end));
         start = end;
         end = bi.next();
      }
   }
 
   public static String readFile(String filename) {
      StringBuffer total = new StringBuffer();
 
      try {
         BufferedReader br = new BufferedReader(new FileReader(filename));
         String line;
         while ((line = br.readLine()) != null) {
            total.append(line + "n");
         }
      }
      catch(Exception e) {
         e.printStackTrace();
         System.exit(1);
      }
      return total.toString();
   }
}

outputs:

Splitting text up into words

Found: It
Found:  
Found: is
Found: ,
Found:  
Found: in
Found:  
Found: fact
Found: ,
Found:  
Found: nothing
Found:  
Found: short
Found:  
Found: of
Found:  
Found: a
Found:  
Found: miracle
Found:  
Found: that
Found:  
Found: the
Found:  
Found: modern
Found:  
Found: methods
Found:  

Found: of
Found:  
Found: instruction
Found:  
Found: have
Found:  
Found: not
Found:  
Found: yet
Found:  
Found: entirely
Found:  
Found: strangled
Found:  
Found: the
Found:  
Found: holy
Found:  
Found: curiosity
Found:  

Found: of
Found:  
Found: inquiry
Found: ;
Found:  
Found: for
. . .

Note that also whitespace and punctuation are returned as being words. You may want to change the method so that it recognizes this fact, for example by checking whether the first character of the word is a letter.