DOC HOME SITE MAP MAN PAGES GNU INFO SEARCH
 

(flex.info.gz) Multiple Input Buffers

Info Catalog (flex.info.gz) Start Conditions (flex.info.gz) Top (flex.info.gz) EOF
 
 11 Multiple Input Buffers
 *************************
 
 Some scanners (such as those which support "include" files) require
 reading from several input streams.  As `flex' scanners do a large
 amount of buffering, one cannot control where the next input will be
 read from by simply writing a `YY_INPUT()' which is sensitive to the
 scanning context.  `YY_INPUT()' is only called when the scanner reaches
 the end of its buffer, which may be a long time after scanning a
 statement such as an `include' statement which requires switching the
 input source.
 
    To negotiate these sorts of problems, `flex' provides a mechanism
 for creating and switching between multiple input buffers.  An input
 buffer is created by using:
 
  -- Function: YY_BUFFER_STATE yy_create_buffer ( FILE *file, int size )
 
    which takes a `FILE' pointer and a size and creates a buffer
 associated with the given file and large enough to hold `size'
 characters (when in doubt, use `YY_BUF_SIZE' for the size).  It returns
 a `YY_BUFFER_STATE' handle, which may then be passed to other routines
 (see below).  The `YY_BUFFER_STATE' type is a pointer to an opaque
 `struct yy_buffer_state' structure, so you may safely initialize
 `YY_BUFFER_STATE' variables to `((YY_BUFFER_STATE) 0)' if you wish, and
 also refer to the opaque structure in order to correctly declare input
 buffers in source files other than that of your scanner.  Note that the
 `FILE' pointer in the call to `yy_create_buffer' is only used as the
 value of `yyin' seen by `YY_INPUT'.  If you redefine `YY_INPUT()' so it
 no longer uses `yyin', then you can safely pass a NULL `FILE' pointer to
 `yy_create_buffer'.  You select a particular buffer to scan from using:
 
  -- Function: void yy_switch_to_buffer ( YY_BUFFER_STATE new_buffer )
 
    The above function switches the scanner's input buffer so subsequent
 tokens will come from `new_buffer'.  Note that `yy_switch_to_buffer()'
 may be used by `yywrap()' to set things up for continued scanning,
 instead of opening a new file and pointing `yyin' at it. If you are
 looking for a stack of input buffers, then you want to use
 `yypush_buffer_state()' instead of this function. Note also that
 switching input sources via either `yy_switch_to_buffer()' or
 `yywrap()' does _not_ change the start condition.
 
  -- Function: void yy_delete_buffer ( YY_BUFFER_STATE buffer )
 
    is used to reclaim the storage associated with a buffer.  (`buffer'
 can be NULL, in which case the routine does nothing.)  You can also
 clear the current contents of a buffer using:
 
  -- Function: void yypush_buffer_state ( YY_BUFFER_STATE buffer )
 
    This function pushes the new buffer state onto an internal stack.
 The pushed state becomes the new current state. The stack is maintained
 by flex and will grow as required. This function is intended to be used
 instead of `yy_switch_to_buffer', when you want to change states, but
 preserve the current state for later use.
 
  -- Function: void yypop_buffer_state ( )
 
    This function removes the current state from the top of the stack,
 and deletes it by calling `yy_delete_buffer'.  The next state on the
 stack, if any, becomes the new current state.
 
  -- Function: void yy_flush_buffer ( YY_BUFFER_STATE buffer )
 
    This function discards the buffer's contents, so the next time the
 scanner attempts to match a token from the buffer, it will first fill
 the buffer anew using `YY_INPUT()'.
 
  -- Function: YY_BUFFER_STATE yy_new_buffer ( FILE *file, int size )
 
    is an alias for `yy_create_buffer()', provided for compatibility
 with the C++ use of `new' and `delete' for creating and destroying
 dynamic objects.
 
    `YY_CURRENT_BUFFER' macro returns a `YY_BUFFER_STATE' handle to the
 current buffer. It should not be used as an lvalue.
 
    Here are two examples of using these features for writing a scanner
 which expands include files (the `<<EOF>>' feature is discussed below).
 
    This first example uses yypush_buffer_state and yypop_buffer_state.
 Flex maintains the stack internally.
 
 
          /* the "incl" state is used for picking up the name
           * of an include file
           */
          %x incl
          %%
          include             BEGIN(incl);
 
          [a-z]+              ECHO;
          [^a-z\n]*\n?        ECHO;
 
          <incl>[ \t]*      /* eat the whitespace */
          <incl>[^ \t\n]+   { /* got the include file name */
                  yyin = fopen( yytext, "r" );
 
                  if ( ! yyin )
                      error( ... );
 
      			yypush_buffer_state(yy_create_buffer( yyin, YY_BUF_SIZE ));
 
                  BEGIN(INITIAL);
                  }
 
          <<EOF>> {
      			yypop_buffer_state();
 
                  if ( !YY_CURRENT_BUFFER )
                      {
                      yyterminate();
                      }
                  }
 
    The second example, below, does the same thing as the previous
 example did, but manages its own input buffer stack manually (instead
 of letting flex do it).
 
 
          /* the "incl" state is used for picking up the name
           * of an include file
           */
          %x incl
 
          %{
          #define MAX_INCLUDE_DEPTH 10
          YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH];
          int include_stack_ptr = 0;
          %}
 
          %%
          include             BEGIN(incl);
 
          [a-z]+              ECHO;
          [^a-z\n]*\n?        ECHO;
 
          <incl>[ \t]*      /* eat the whitespace */
          <incl>[^ \t\n]+   { /* got the include file name */
                  if ( include_stack_ptr >= MAX_INCLUDE_DEPTH )
                      {
                      fprintf( stderr, "Includes nested too deeply" );
                      exit( 1 );
                      }
 
                  include_stack[include_stack_ptr++] =
                      YY_CURRENT_BUFFER;
 
                  yyin = fopen( yytext, "r" );
 
                  if ( ! yyin )
                      error( ... );
 
                  yy_switch_to_buffer(
                      yy_create_buffer( yyin, YY_BUF_SIZE ) );
 
                  BEGIN(INITIAL);
                  }
 
          <<EOF>> {
                  if ( --include_stack_ptr  0 )
                      {
                      yyterminate();
                      }
 
                  else
                      {
                      yy_delete_buffer( YY_CURRENT_BUFFER );
                      yy_switch_to_buffer(
                           include_stack[include_stack_ptr] );
                      }
                  }
 
    The following routines are available for setting up input buffers for
 scanning in-memory strings instead of files.  All of them create a new
 input buffer for scanning the string, and return a corresponding
 `YY_BUFFER_STATE' handle (which you should delete with
 `yy_delete_buffer()' when done with it).  They also switch to the new
 buffer using `yy_switch_to_buffer()', so the next call to `yylex()'
 will start scanning the string.
 
  -- Function: YY_BUFFER_STATE yy_scan_string ( const char *str )
      scans a NUL-terminated string.
 
  -- Function: YY_BUFFER_STATE yy_scan_bytes ( const char *bytes, int
           len )
      scans `len' bytes (including possibly `NUL's) starting at location
      `bytes'.
 
    Note that both of these functions create and scan a _copy_ of the
 string or bytes.  (This may be desirable, since `yylex()' modifies the
 contents of the buffer it is scanning.)  You can avoid the copy by
 using:
 
  -- Function: YY_BUFFER_STATE yy_scan_buffer (char *base, yy_size_t
           size)
      which scans in place the buffer starting at `base', consisting of
      `size' bytes, the last two bytes of which _must_ be
      `YY_END_OF_BUFFER_CHAR' (ASCII NUL).  These last two bytes are not
      scanned; thus, scanning consists of `base[0]' through
      `base[size-2]', inclusive.
 
    If you fail to set up `base' in this manner (i.e., forget the final
 two `YY_END_OF_BUFFER_CHAR' bytes), then `yy_scan_buffer()' returns a
 NULL pointer instead of creating a new input buffer.
 
  -- Data type: yy_size_t
      is an integral type to which you can cast an integer expression
      reflecting the size of the buffer.
 
Info Catalog (flex.info.gz) Start Conditions (flex.info.gz) Top (flex.info.gz) EOF
automatically generated byinfo2html