OpenMD  2.5
Molecular Dynamics in the Open
antlr::TokenBuffer Class Reference

#include <TokenBuffer.hpp>

Public Member Functions

 TokenBuffer (TokenStream &input_)
 
virtual ~TokenBuffer ()
 
void reset (void)
 Reset the input buffer to empty state. More...
 
int LA (unsigned int i)
 
RefToken LT (unsigned int i)
 
unsigned int mark ()
 
void rewind (unsigned int mark)
 
void consume ()
 
virtual unsigned int entries () const
 Return the number of entries in the TokenBuffer. More...
 

Protected Attributes

TokenStreaminput
 Token source. More...
 
unsigned int nMarkers
 Number of active markers. More...
 
unsigned int markerOffset
 Additional offset used when markers are active. More...
 
unsigned int numToConsume
 Number of calls to consume() since last LA() or LT() call. More...
 
CircularQueue< RefTokenqueue
 Circular queue with Tokens. More...
 

Private Member Functions

void fill (unsigned int amount)
 
void syncConsume ()
 
 TokenBuffer (const TokenBuffer &other)
 
const TokenBufferoperator= (const TokenBuffer &other)
 

Detailed Description

A Stream of Token objects fed to the parser from a TokenStream that can be rewound via mark()/rewind() methods.

A dynamic array is used to buffer up all the input tokens. Normally, "k" tokens are stored in the buffer. More tokens may be stored during guess mode (testing syntactic predicate), or when LT(i>k) is referenced. Consumption of tokens is deferred. In other words, reading the next token is not done by conume(), but deferred until needed by LA or LT.

Todo:
: see if we can integrate this one with InputBuffer into one template or so.
See also
antlr.Token
antlr.TokenStream
antlr.TokenQueue

Definition at line 36 of file TokenBuffer.hpp.

Constructor & Destructor Documentation

◆ TokenBuffer() [1/2]

antlr::TokenBuffer::TokenBuffer ( TokenStream inp)

Create a token buffer

A Stream of Token objects fed to the parser from a TokenStream that can be rewound via mark()/rewind() methods.

A dynamic array is used to buffer up all the input tokens. Normally, "k" tokens are stored in the buffer. More tokens may be stored during guess mode (testing syntactic predicate), or when LT(i>k) is referenced. Consumption of tokens is deferred. In other words, reading the next token is not done by conume(), but deferred until needed by LA or LT.

See also
antlr.Token
antlr.TokenStream
antlr.TokenQueueCreate a token buffer

Definition at line 30 of file TokenBuffer.cpp.

◆ ~TokenBuffer()

antlr::TokenBuffer::~TokenBuffer ( void  )
virtual

Definition at line 38 of file TokenBuffer.cpp.

◆ TokenBuffer() [2/2]

antlr::TokenBuffer::TokenBuffer ( const TokenBuffer other)
private

Member Function Documentation

◆ consume()

void antlr::TokenBuffer::consume ( )
inline

Mark another token for deferred consumption

Definition at line 68 of file TokenBuffer.hpp.

◆ entries()

unsigned int antlr::TokenBuffer::entries ( ) const
virtual

Return the number of entries in the TokenBuffer.

Get number of non-consumed tokens.

Definition at line 89 of file TokenBuffer.cpp.

References markerOffset, and queue.

◆ fill()

void antlr::TokenBuffer::fill ( unsigned int  amount)
private

Ensure that the token buffer is sufficiently full

Definition at line 43 of file TokenBuffer.cpp.

References input, markerOffset, antlr::TokenStream::nextToken(), queue, and syncConsume().

Referenced by LA(), and LT().

◆ LA()

int antlr::TokenBuffer::LA ( unsigned int  i)

Get a lookahead token value

Definition at line 55 of file TokenBuffer.cpp.

References fill(), markerOffset, and queue.

◆ LT()

RefToken antlr::TokenBuffer::LT ( unsigned int  i)

Get a lookahead token

Definition at line 62 of file TokenBuffer.cpp.

References fill(), markerOffset, and queue.

◆ mark()

unsigned int antlr::TokenBuffer::mark ( )

Return an integer marker that can be used to rewind the buffer to its current state.

Definition at line 71 of file TokenBuffer.cpp.

References markerOffset, nMarkers, and syncConsume().

Referenced by rewind().

◆ operator=()

const TokenBuffer& antlr::TokenBuffer::operator= ( const TokenBuffer other)
private

◆ reset()

void antlr::TokenBuffer::reset ( void  )
inline

Reset the input buffer to empty state.

Definition at line 43 of file TokenBuffer.hpp.

◆ rewind()

void antlr::TokenBuffer::rewind ( unsigned int  mark)

Rewind the token buffer to a marker.

Parameters
markMarker returned previously from mark()

Definition at line 81 of file TokenBuffer.cpp.

References mark(), markerOffset, nMarkers, and syncConsume().

◆ syncConsume()

void antlr::TokenBuffer::syncConsume ( )
inlineprivate

Sync up deferred consumption

Definition at line 104 of file TokenBuffer.hpp.

Referenced by fill(), mark(), and rewind().

Member Data Documentation

◆ input

TokenStream& antlr::TokenBuffer::input
protected

Token source.

Definition at line 84 of file TokenBuffer.hpp.

Referenced by fill().

◆ markerOffset

unsigned int antlr::TokenBuffer::markerOffset
protected

Additional offset used when markers are active.

Definition at line 90 of file TokenBuffer.hpp.

Referenced by entries(), fill(), LA(), LT(), mark(), and rewind().

◆ nMarkers

unsigned int antlr::TokenBuffer::nMarkers
protected

Number of active markers.

Definition at line 87 of file TokenBuffer.hpp.

Referenced by mark(), and rewind().

◆ numToConsume

unsigned int antlr::TokenBuffer::numToConsume
protected

Number of calls to consume() since last LA() or LT() call.

Definition at line 93 of file TokenBuffer.hpp.

◆ queue

CircularQueue<RefToken> antlr::TokenBuffer::queue
protected

Circular queue with Tokens.

Definition at line 96 of file TokenBuffer.hpp.

Referenced by entries(), fill(), LA(), and LT().


The documentation for this class was generated from the following files: