Large Files

• Need to parse large (>100MB) files

• mxTextTools is in memory

  • can't even use mmap'ed files
  • parse tree is also in memory

• Luckily

  • large files are built of small records
  • records can be identified without full processing

• Decided to use a two-stage parser

Notes: