antlr::TokenBuffer Class Reference

A Stream of Token objects fed to the parser from a TokenStream that can be rewound via mark()/rewind() methods. More...

#include <TokenBuffer.hpp>

Public Member Functions
	TokenBuffer (TokenStream &input_)
	Create a token buffer.
void	reset (void)
	Reset the input buffer to empty state.
int	LA (unsigned int i)
	Get a lookahead token value.
RefToken	LT (unsigned int i)
	Get a lookahead token.
unsigned int	mark ()
	Return an integer marker that can be used to rewind the buffer to its current state.
void	rewind (unsigned int mark)
	Rewind the token buffer to a marker.
void	consume ()
	Mark another token for deferred consumption.
virtual unsigned int	entries () const
	Return the number of entries in the TokenBuffer.

Protected Attributes
TokenStream &	input
	Token source.
unsigned int	nMarkers
	Number of active markers.
unsigned int	markerOffset
	Additional offset used when markers are active.
unsigned int	numToConsume
	Number of calls to consume() since last LA() or LT() call.
CircularQueue< RefToken >	queue
	Circular queue with Tokens.

Detailed Description

A Stream of Token objects fed to the parser from a TokenStream that can be rewound via mark()/rewind() methods.

A dynamic array is used to buffer up all the input tokens. Normally, "k" tokens are stored in the buffer. More tokens may be stored during guess mode (testing syntactic predicate), or when LT(i>k) is referenced. Consumption of tokens is deferred. In other words, reading the next token is not done by conume(), but deferred until needed by LA or LT.

Todo

: see if we can integrate this one with InputBuffer into one template or so.

See also

antlr.Token

antlr.TokenStream

antlr.TokenQueue

Definition at line 36 of file TokenBuffer.hpp.

Constructor & Destructor Documentation

TokenBuffer()

antlr::TokenBuffer::TokenBuffer

(

TokenStream &

inp

)

Create a token buffer.

A Stream of Token objects fed to the parser from a TokenStream that can be rewound via mark()/rewind() methods.

A dynamic array is used to buffer up all the input tokens. Normally, "k" tokens are stored in the buffer. More tokens may be stored during guess mode (testing syntactic predicate), or when LT(i>k) is referenced. Consumption of tokens is deferred. In other words, reading the next token is not done by conume(), but deferred until needed by LA or LT.

See also

antlr.Token

antlr.TokenStream

antlr.TokenQueue Create a token buffer

Definition at line 30 of file TokenBuffer.cpp.

~TokenBuffer()

antlr::TokenBuffer::~TokenBuffer ( void )

virtual

Definition at line 38 of file TokenBuffer.cpp.

Member Function Documentation

consume()

void antlr::TokenBuffer::consume ( )

inline

Mark another token for deferred consumption.

Definition at line 68 of file TokenBuffer.hpp.

entries()

unsigned int antlr::TokenBuffer::entries ( ) const

virtual

Return the number of entries in the TokenBuffer.

Get number of non-consumed tokens.

Definition at line 89 of file TokenBuffer.cpp.

References markerOffset.

LA()

int antlr::TokenBuffer::LA

(

unsigned int

i

)

Get a lookahead token value.

Definition at line 55 of file TokenBuffer.cpp.

References markerOffset.

LT()

RefToken antlr::TokenBuffer::LT

(

unsigned int

i

)

Get a lookahead token.

Definition at line 62 of file TokenBuffer.cpp.

References markerOffset.

mark()

unsigned int antlr::TokenBuffer::mark

(

)

Return an integer marker that can be used to rewind the buffer to its current state.

Definition at line 71 of file TokenBuffer.cpp.

References markerOffset, and nMarkers.

Referenced by rewind().

reset()

void antlr::TokenBuffer::reset ( void )

inline

Reset the input buffer to empty state.

Definition at line 43 of file TokenBuffer.hpp.

rewind()

void antlr::TokenBuffer::rewind

(

unsigned int

mark

)

Rewind the token buffer to a marker.

Parameters

mark	Marker returned previously from mark()

Definition at line 81 of file TokenBuffer.cpp.

References mark(), markerOffset, and nMarkers.

Member Data Documentation

input

TokenStream& antlr::TokenBuffer::input

protected

Token source.

Definition at line 84 of file TokenBuffer.hpp.

markerOffset

unsigned int antlr::TokenBuffer::markerOffset

protected

Additional offset used when markers are active.

Definition at line 90 of file TokenBuffer.hpp.

Referenced by entries(), LA(), LT(), mark(), and rewind().

nMarkers

unsigned int antlr::TokenBuffer::nMarkers

protected

Number of active markers.

Definition at line 87 of file TokenBuffer.hpp.

Referenced by mark(), and rewind().

numToConsume

unsigned int antlr::TokenBuffer::numToConsume

protected

Number of calls to consume() since last LA() or LT() call.

Definition at line 93 of file TokenBuffer.hpp.

queue

CircularQueue<RefToken> antlr::TokenBuffer::queue

protected

Circular queue with Tokens.

Definition at line 96 of file TokenBuffer.hpp.

---

The documentation for this class was generated from the following files:

• antlr/TokenBuffer.hpp
• antlr/TokenBuffer.cpp