This token stream tracks the entire token stream coming from a lexer, but does not pass on the whitespace (or whatever else you want to discard) to the parser. More...

#include <TokenStreamRewriteEngine.hpp>

Inheritance diagram for antlr::TokenStreamRewriteEngine:

Classes
class	DeleteOp

struct	executeOperation

class	InsertBeforeOp

class	ReplaceOp

class	RewriteOperation

struct	tokenToStream

Public Types
typedef std ::vector< antlr::RefTokenWithIndex >	token_list

typedef std ::list< RewriteOperation * >	operation_list
	list of rewrite operations

typedef std ::map< std ::string, operation_list >	program_map
	map program name to <program counter,program> tuple

Public Member Functions
	TokenStreamRewriteEngine (TokenStream &upstream)

	TokenStreamRewriteEngine (TokenStream &upstream, size_t initialSize)

RefToken	nextToken (void)

void	rollback (size_t instructionIndex)

void	rollback (const std ::string &programName, size_t instructionIndex)
	Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream.

void	deleteProgram ()

void	deleteProgram (const std ::string &programName)
	Reset the program so that no instructions exist.

void	insertAfter (RefTokenWithIndex t, const std ::string &text)

void	insertAfter (size_t index, const std ::string &text)

void	insertAfter (const std ::string &programName, RefTokenWithIndex t, const std ::string &text)

void	insertAfter (const std ::string &programName, size_t index, const std ::string &text)

void	insertBefore (RefTokenWithIndex t, const std ::string &text)

void	insertBefore (size_t index, const std ::string &text)

void	insertBefore (const std ::string &programName, RefTokenWithIndex t, const std ::string &text)

void	insertBefore (const std ::string &programName, size_t index, const std ::string &text)

void	replace (size_t index, const std ::string &text)

void	replace (size_t from, size_t to, const std ::string &text)

void	replace (RefTokenWithIndex indexT, const std ::string &text)

void	replace (RefTokenWithIndex from, RefTokenWithIndex to, const std ::string &text)

void	replace (const std ::string &programName, size_t from, size_t to, const std ::string &text)

void	replace (const std ::string &programName, RefTokenWithIndex from, RefTokenWithIndex to, const std ::string &text)

void	remove (size_t index)

void	remove (size_t from, size_t to)

void	remove (RefTokenWithIndex indexT)

void	remove (RefTokenWithIndex from, RefTokenWithIndex to)

void	remove (const std ::string &programName, size_t from, size_t to)

void	remove (const std ::string &programName, RefTokenWithIndex from, RefTokenWithIndex to)

void	discard (int ttype)

RefToken	getToken (size_t i)

size_t	getTokenStreamSize () const

void	originalToStream (std ::ostream &out) const

void	originalToStream (std ::ostream &out, size_t start, size_t end) const

void	toStream (std ::ostream &out) const

void	toStream (std ::ostream &out, const std ::string &programName) const

void	toStream (std ::ostream &out, size_t start, size_t end) const

void	toStream (std ::ostream &out, const std ::string &programName, size_t firstToken, size_t lastToken) const

void	toDebugStream (std ::ostream &out) const

void	toDebugStream (std ::ostream &out, size_t start, size_t end) const

size_t	getLastRewriteTokenIndex () const

size_t	getLastRewriteTokenIndex (const std ::string &programName) const
	Return the last index for the program named programName return 0 if the program does not exist or the program is empty.

Static Public Attributes
static const char *	DEFAULT_PROGRAM_NAME = "default"

static const size_t	MIN_TOKEN_INDEX = 0

static const int	PROGRAM_INIT_SIZE = 100

Protected Member Functions
void	addToSortedRewriteList (RewriteOperation *op)
	If op.index > lastRewriteTokenIndexes, just add to the end.

void	addToSortedRewriteList (const std ::string &programName, RewriteOperation *op)

Protected Attributes
TokenStream &	stream
	Who do we suck tokens from?

size_t	index
	track index of tokens

token_list	tokens
	Track the incoming list of tokens.

program_map	programs
	You may have multiple, named streams of rewrite operations.

BitSet	discardMask
	Which (whitespace) token(s) to throw out.

Detailed Description

This token stream tracks the entire token stream coming from a lexer, but does not pass on the whitespace (or whatever else you want to discard) to the parser.

This class can then be asked for the ith token in the input stream. Useful for dumping out the input stream exactly after doing some augmentation or other manipulations. Tokens are index from 0..n-1

You can insert stuff, replace, and delete chunks. Note that the operations are done lazily–only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :)

Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1.

Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example,

       TokenStreamRewriteEngine rewriteEngine =
               new TokenStreamRewriteEngine(lexer);
 JavaRecognizer parser = new JavaRecognizer(rewriteEngine);
 ...
 rewriteEngine.insertAfter("pass1", t, "foobar");}
       rewriteEngine.insertAfter("pass2", u, "start");}
       System.out.println(rewriteEngine.toString("pass1"));
       System.out.println(rewriteEngine.toString("pass2"));

You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file–all from the same buffer.

If you don't use named rewrite streams, a "default" stream is used.

Terence Parr, parrt.nosp@m.@cs..nosp@m.usfca.nosp@m..edu University of San Francisco February 2004

Definition at line 77 of file TokenStreamRewriteEngine.hpp.

Member Typedef Documentation

◆ operation_list

typedef std :: list<RewriteOperation*> antlr::TokenStreamRewriteEngine::operation_list

list of rewrite operations

Definition at line 136 of file TokenStreamRewriteEngine.hpp.

◆ program_map

typedef std :: map< std :: string,operation_list> antlr::TokenStreamRewriteEngine::program_map

map program name to <program counter,program> tuple

Definition at line 138 of file TokenStreamRewriteEngine.hpp.

◆ token_list

typedef std :: vector<antlr::RefTokenWithIndex> antlr::TokenStreamRewriteEngine::token_list

Definition at line 80 of file TokenStreamRewriteEngine.hpp.

Constructor & Destructor Documentation

◆ TokenStreamRewriteEngine() [1/2]

antlr::TokenStreamRewriteEngine::TokenStreamRewriteEngine ( TokenStream & upstream )

Definition at line 47 of file TokenStreamRewriteEngine.cpp.

◆ TokenStreamRewriteEngine() [2/2]

antlr::TokenStreamRewriteEngine::TokenStreamRewriteEngine	(	TokenStream &	upstream,
		size_t	initialSize )

Definition at line 56 of file TokenStreamRewriteEngine.cpp.

Member Function Documentation

◆ addToSortedRewriteList()

void antlr::TokenStreamRewriteEngine::addToSortedRewriteList ( RewriteOperation * op )

inlineprotected

If op.index > lastRewriteTokenIndexes, just add to the end.

Otherwise, do linear

Definition at line 409 of file TokenStreamRewriteEngine.hpp.

References addToSortedRewriteList().

Referenced by addToSortedRewriteList().

◆ deleteProgram() [1/2]

void antlr::TokenStreamRewriteEngine::deleteProgram ( )

inline

Definition at line 206 of file TokenStreamRewriteEngine.hpp.

◆ deleteProgram() [2/2]

void antlr::TokenStreamRewriteEngine::deleteProgram ( const std ::string & programName )

inline

Reset the program so that no instructions exist.

Definition at line 211 of file TokenStreamRewriteEngine.hpp.

◆ discard()

void antlr::TokenStreamRewriteEngine::discard ( int ttype )

inline

Definition at line 335 of file TokenStreamRewriteEngine.hpp.

◆ getLastRewriteTokenIndex() [1/2]

size_t antlr::TokenStreamRewriteEngine::getLastRewriteTokenIndex ( ) const

inline

Definition at line 382 of file TokenStreamRewriteEngine.hpp.

◆ getLastRewriteTokenIndex() [2/2]

size_t antlr::TokenStreamRewriteEngine::getLastRewriteTokenIndex ( const std ::string & programName ) const

inline

Return the last index for the program named programName return 0 if the program does not exist or the program is empty.

(Note this is different from the java implementation that returns -1)

Definition at line 390 of file TokenStreamRewriteEngine.hpp.

References programs.

◆ getToken()

RefToken antlr::TokenStreamRewriteEngine::getToken ( size_t i )

inline

Definition at line 339 of file TokenStreamRewriteEngine.hpp.

◆ getTokenStreamSize()

size_t antlr::TokenStreamRewriteEngine::getTokenStreamSize ( ) const

inline

Definition at line 344 of file TokenStreamRewriteEngine.hpp.

◆ insertAfter() [1/4]

void antlr::TokenStreamRewriteEngine::insertAfter	(	const std ::string &	programName,
		RefTokenWithIndex	t,
		const std ::string &	text )

inline

Definition at line 225 of file TokenStreamRewriteEngine.hpp.

◆ insertAfter() [2/4]

void antlr::TokenStreamRewriteEngine::insertAfter	(	const std ::string &	programName,
		size_t	index,
		const std ::string &	text )

inline

Definition at line 232 of file TokenStreamRewriteEngine.hpp.

◆ insertAfter() [3/4]

void antlr::TokenStreamRewriteEngine::insertAfter	(	RefTokenWithIndex	t,
		const std ::string &	text )

inline

Definition at line 215 of file TokenStreamRewriteEngine.hpp.

◆ insertAfter() [4/4]

void antlr::TokenStreamRewriteEngine::insertAfter	(	size_t	index,
		const std ::string &	text )

inline

Definition at line 221 of file TokenStreamRewriteEngine.hpp.

◆ insertBefore() [1/4]

void antlr::TokenStreamRewriteEngine::insertBefore	(	const std ::string &	programName,
		RefTokenWithIndex	t,
		const std ::string &	text )

inline

Definition at line 251 of file TokenStreamRewriteEngine.hpp.

◆ insertBefore() [2/4]

void antlr::TokenStreamRewriteEngine::insertBefore	(	const std ::string &	programName,
		size_t	index,
		const std ::string &	text )

inline

Definition at line 258 of file TokenStreamRewriteEngine.hpp.

◆ insertBefore() [3/4]

void antlr::TokenStreamRewriteEngine::insertBefore	(	RefTokenWithIndex	t,
		const std ::string &	text )

inline

Definition at line 240 of file TokenStreamRewriteEngine.hpp.

◆ insertBefore() [4/4]

void antlr::TokenStreamRewriteEngine::insertBefore	(	size_t	index,
		const std ::string &	text )

inline

Definition at line 247 of file TokenStreamRewriteEngine.hpp.

◆ nextToken()

RefToken antlr::TokenStreamRewriteEngine::nextToken ( void )

virtual

Implements antlr::TokenStream.

Definition at line 65 of file TokenStreamRewriteEngine.cpp.

◆ originalToStream()

void antlr::TokenStreamRewriteEngine::originalToStream ( std ::ostream & out ) const

inline

Definition at line 348 of file TokenStreamRewriteEngine.hpp.

◆ remove() [1/6]

void antlr::TokenStreamRewriteEngine::remove	(	const std ::string &	programName,
		RefTokenWithIndex	from,
		RefTokenWithIndex	to )

inline

Definition at line 329 of file TokenStreamRewriteEngine.hpp.

◆ remove() [2/6]

void antlr::TokenStreamRewriteEngine::remove	(	const std ::string &	programName,
		size_t	from,
		size_t	to )

inline

Definition at line 323 of file TokenStreamRewriteEngine.hpp.

◆ remove() [3/6]

void antlr::TokenStreamRewriteEngine::remove	(	RefTokenWithIndex	from,
		RefTokenWithIndex	to )

inline

Definition at line 319 of file TokenStreamRewriteEngine.hpp.

◆ remove() [4/6]

void antlr::TokenStreamRewriteEngine::remove ( RefTokenWithIndex indexT )

inline

Definition at line 315 of file TokenStreamRewriteEngine.hpp.

◆ remove() [5/6]

void antlr::TokenStreamRewriteEngine::remove	(	size_t	from,
		size_t	to )

inline

Definition at line 311 of file TokenStreamRewriteEngine.hpp.

◆ remove() [6/6]

void antlr::TokenStreamRewriteEngine::remove ( size_t index )

inline

Definition at line 307 of file TokenStreamRewriteEngine.hpp.

◆ replace() [1/6]

void antlr::TokenStreamRewriteEngine::replace	(	const std ::string &	programName,
		RefTokenWithIndex	from,
		RefTokenWithIndex	to,
		const std ::string &	text )

inline

Definition at line 296 of file TokenStreamRewriteEngine.hpp.

◆ replace() [2/6]

void antlr::TokenStreamRewriteEngine::replace	(	const std ::string &	programName,
		size_t	from,
		size_t	to,
		const std ::string &	text )

inline

Definition at line 289 of file TokenStreamRewriteEngine.hpp.

◆ replace() [3/6]

void antlr::TokenStreamRewriteEngine::replace	(	RefTokenWithIndex	from,
		RefTokenWithIndex	to,
		const std ::string &	text )

inline

Definition at line 282 of file TokenStreamRewriteEngine.hpp.

◆ replace() [4/6]

void antlr::TokenStreamRewriteEngine::replace	(	RefTokenWithIndex	indexT,
		const std ::string &	text )

inline

Definition at line 276 of file TokenStreamRewriteEngine.hpp.

◆ replace() [5/6]

void antlr::TokenStreamRewriteEngine::replace	(	size_t	from,
		size_t	to,
		const std ::string &	text )

inline

Definition at line 270 of file TokenStreamRewriteEngine.hpp.

◆ replace() [6/6]

void antlr::TokenStreamRewriteEngine::replace	(	size_t	index,
		const std ::string &	text )

inline

Definition at line 265 of file TokenStreamRewriteEngine.hpp.

◆ rollback() [1/2]

void antlr::TokenStreamRewriteEngine::rollback	(	const std ::string &	programName,
		size_t	instructionIndex )

Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream.

UNTESTED!

◆ rollback() [2/2]

void antlr::TokenStreamRewriteEngine::rollback ( size_t instructionIndex )

inline

Definition at line 195 of file TokenStreamRewriteEngine.hpp.

◆ toDebugStream()

void antlr::TokenStreamRewriteEngine::toDebugStream ( std ::ostream & out ) const

inline

Definition at line 375 of file TokenStreamRewriteEngine.hpp.

◆ toStream() [1/3]

void antlr::TokenStreamRewriteEngine::toStream ( std ::ostream & out ) const

inline

Definition at line 355 of file TokenStreamRewriteEngine.hpp.

◆ toStream() [2/3]

void antlr::TokenStreamRewriteEngine::toStream	(	std ::ostream &	out,
		const std ::string &	programName ) const

inline

Definition at line 359 of file TokenStreamRewriteEngine.hpp.

◆ toStream() [3/3]

void antlr::TokenStreamRewriteEngine::toStream	(	std ::ostream &	out,
		size_t	start,
		size_t	end ) const

inline

Definition at line 365 of file TokenStreamRewriteEngine.hpp.

Member Data Documentation

◆ DEFAULT_PROGRAM_NAME

const char * antlr::TokenStreamRewriteEngine::DEFAULT_PROGRAM_NAME = "default"

static

Definition at line 81 of file TokenStreamRewriteEngine.hpp.

◆ discardMask

BitSet antlr::TokenStreamRewriteEngine::discardMask

protected

Which (whitespace) token(s) to throw out.

Definition at line 432 of file TokenStreamRewriteEngine.hpp.

◆ index

size_t antlr::TokenStreamRewriteEngine::index

protected

track index of tokens

Definition at line 420 of file TokenStreamRewriteEngine.hpp.

◆ MIN_TOKEN_INDEX

const size_t antlr::TokenStreamRewriteEngine::MIN_TOKEN_INDEX = 0

static

Definition at line 83 of file TokenStreamRewriteEngine.hpp.

◆ PROGRAM_INIT_SIZE

const int antlr::TokenStreamRewriteEngine::PROGRAM_INIT_SIZE = 100

static

Definition at line 84 of file TokenStreamRewriteEngine.hpp.

◆ programs

program_map antlr::TokenStreamRewriteEngine::programs

protected

You may have multiple, named streams of rewrite operations.

I'm calling these things "programs." Maps String (name) -> rewrite (List)

Definition at line 429 of file TokenStreamRewriteEngine.hpp.

Referenced by getLastRewriteTokenIndex().

◆ stream

TokenStream& antlr::TokenStreamRewriteEngine::stream

protected

Who do we suck tokens from?

Definition at line 418 of file TokenStreamRewriteEngine.hpp.

◆ tokens

token_list antlr::TokenStreamRewriteEngine::tokens

protected

Track the incoming list of tokens.

Definition at line 423 of file TokenStreamRewriteEngine.hpp.

The documentation for this class was generated from the following files:

antlr/TokenStreamRewriteEngine.hpp
antlr/TokenStreamRewriteEngine.cpp

Classes

Public Types

Public Member Functions

Static Public Attributes

Protected Member Functions

Protected Attributes

Detailed Description

Member Typedef Documentation

◆ operation_list

◆ program_map

◆ token_list

Constructor & Destructor Documentation

◆ TokenStreamRewriteEngine() [1/2]

◆ TokenStreamRewriteEngine() [2/2]

Member Function Documentation

◆ addToSortedRewriteList()

◆ deleteProgram() [1/2]

◆ deleteProgram() [2/2]

◆ discard()

◆ getLastRewriteTokenIndex() [1/2]

◆ getLastRewriteTokenIndex() [2/2]

◆ getToken()

◆ getTokenStreamSize()

◆ insertAfter() [1/4]

◆ insertAfter() [2/4]

◆ insertAfter() [3/4]

◆ insertAfter() [4/4]

◆ insertBefore() [1/4]

◆ insertBefore() [2/4]

◆ insertBefore() [3/4]

◆ insertBefore() [4/4]

◆ nextToken()

◆ originalToStream()

◆ remove() [1/6]

◆ remove() [2/6]

◆ remove() [3/6]

◆ remove() [4/6]

◆ remove() [5/6]

◆ remove() [6/6]

◆ replace() [1/6]

◆ replace() [2/6]

◆ replace() [3/6]

◆ replace() [4/6]

◆ replace() [5/6]

◆ replace() [6/6]

◆ rollback() [1/2]

◆ rollback() [2/2]

◆ toDebugStream()

◆ toStream() [1/3]

◆ toStream() [2/3]

◆ toStream() [3/3]

Member Data Documentation

◆ DEFAULT_PROGRAM_NAME

◆ discardMask

◆ index

◆ MIN_TOKEN_INDEX

◆ PROGRAM_INIT_SIZE

◆ programs

◆ stream

◆ tokens