Class: EBNF::LL1::Scanner
- Defined in:
- vendor/bundler/ruby/3.3.0/bundler/gems/ebnf-c8f40958c6c3/lib/ebnf/ll1/scanner.rb
Overview
Overload StringScanner with file operations and line counting
- Reloads scanner as required until EOF.
- Loads to a high-water and reloads when remaining size reaches a low-water.
FIXME: Only implements the subset required by the Lexer for now.
Constant Summary collapse
- HIGH_WATER =
Hopefully large enough to deal with long multi-line comments
512 * 1024
- LOW_WATER =
4 * 1024
Instance Attribute Summary collapse
- #input ⇒ String, ... readonly
-
#lineno ⇒ Integer
The current line number (one-based).
Instance Method Summary collapse
-
#ensure_buffer_full ⇒ Object
Ensures that the input buffer is full to the high water mark, or end of file.
-
#eos? ⇒ Boolean
Returns true if the scan pointer is at the end of the string.
-
#initialize(input, **options) ⇒ Scanner
constructor
Create a scanner, from an IO.
-
#rest ⇒ String
Returns the "rest" of the line, or the next line if at EOL (i.e. everything after the scan pointer).
-
#scan(pattern) ⇒ String
Tries to match with
pattern
at the current position. -
#scan_until(pattern) ⇒ String
Scans the string until the pattern is matched.
-
#skip(pattern) ⇒ Object
Attempts to skip over the given
pattern
beginning with the scan pointer. -
#skip_until(pattern) ⇒ Object
Advances the scan pointer until pattern is matched and consumed.
-
#terminate ⇒ Object
Set the scan pointer to the end of the string and clear matching data.
-
#unscan ⇒ Object
Sets the scan pointer to the previous position.
Constructor Details
#initialize(input, **options) ⇒ Scanner
Create a scanner, from an IO
34 35 36 37 38 39 40 41 42 |
# File 'vendor/bundler/ruby/3.3.0/bundler/gems/ebnf-c8f40958c6c3/lib/ebnf/ll1/scanner.rb', line 34 def initialize(input, **) @options = .merge(high_water: HIGH_WATER, low_water: LOW_WATER) @previous_lineno = @lineno = 1 @input = input.is_a?(String) ? encode_utf8(input) : input super(input.is_a?(String) ? @input : "") feed_me self end |
Instance Attribute Details
#input ⇒ String, ... (readonly)
18 19 20 |
# File 'vendor/bundler/ruby/3.3.0/bundler/gems/ebnf-c8f40958c6c3/lib/ebnf/ll1/scanner.rb', line 18 def input @input end |
#lineno ⇒ Integer
The current line number (one-based).
24 25 26 |
# File 'vendor/bundler/ruby/3.3.0/bundler/gems/ebnf-c8f40958c6c3/lib/ebnf/ll1/scanner.rb', line 24 def lineno @lineno end |
Instance Method Details
#ensure_buffer_full ⇒ Object
Ensures that the input buffer is full to the high water mark, or end of file. Useful when matching tokens that may be longer than the low water mark
46 47 48 49 50 51 52 53 54 |
# File 'vendor/bundler/ruby/3.3.0/bundler/gems/ebnf-c8f40958c6c3/lib/ebnf/ll1/scanner.rb', line 46 def ensure_buffer_full # Read up to high-water mark ensuring we're at an end of line if @input.respond_to?(:eof?) && !@input.eof? diff = @options[:high_water] - rest_size string = encode_utf8(@input.read(diff)) string << encode_utf8(@input.gets) unless @input.eof? self << string if string end end |
#eos? ⇒ Boolean
Returns true if the scan pointer is at the end of the string
60 61 62 63 |
# File 'vendor/bundler/ruby/3.3.0/bundler/gems/ebnf-c8f40958c6c3/lib/ebnf/ll1/scanner.rb', line 60 def eos? feed_me super end |
#rest ⇒ String
Returns the "rest" of the line, or the next line if at EOL (i.e. everything after the scan pointer). If there is no more data (eos? = true), it returns "".
70 71 72 73 |
# File 'vendor/bundler/ruby/3.3.0/bundler/gems/ebnf-c8f40958c6c3/lib/ebnf/ll1/scanner.rb', line 70 def rest feed_me encode_utf8 super end |
#scan(pattern) ⇒ String
Tries to match with pattern
at the current position.
If there is a match, the scanner advances the "scan pointer" and returns the matched string. Otherwise, the scanner returns nil.
If the scanner begins with the multi-line start expression
92 93 94 95 96 97 98 99 |
# File 'vendor/bundler/ruby/3.3.0/bundler/gems/ebnf-c8f40958c6c3/lib/ebnf/ll1/scanner.rb', line 92 def scan(pattern) feed_me @previous_lineno = @lineno if matched = encode_utf8(super) @lineno += matched.count("\n") end matched end |
#scan_until(pattern) ⇒ String
Scans the string until the pattern is matched. Returns the substring up to and including the end of the match, advancing the scan pointer to that location. If there is no match, nil is returned.
112 113 114 115 116 117 118 119 |
# File 'vendor/bundler/ruby/3.3.0/bundler/gems/ebnf-c8f40958c6c3/lib/ebnf/ll1/scanner.rb', line 112 def scan_until(pattern) feed_me @previous_lineno = @lineno if matched = encode_utf8(super) @lineno += matched.count("\n") end matched end |
#skip(pattern) ⇒ Object
Attempts to skip over the given pattern
beginning with the scan pointer.
If it matches, the scan pointer is advanced to the end of the match,
and the length of the match is returned. Otherwise, nil
is returned.
similar to scan
, but without returning the matched string.
128 129 130 131 |
# File 'vendor/bundler/ruby/3.3.0/bundler/gems/ebnf-c8f40958c6c3/lib/ebnf/ll1/scanner.rb', line 128 def skip(pattern) scan(pattern) nil end |
#skip_until(pattern) ⇒ Object
Advances the scan pointer until pattern is matched and consumed. Returns the number of bytes advanced, or nil if no match was found.
Look ahead to match pattern, and advance the scan pointer to the end of the match. Return the number of characters advanced, or nil if the match was unsuccessful.
It’s similar to scan_until, but without returning the intervening string.
140 141 142 |
# File 'vendor/bundler/ruby/3.3.0/bundler/gems/ebnf-c8f40958c6c3/lib/ebnf/ll1/scanner.rb', line 140 def skip_until(pattern) (matched = scan_until(pattern)) && matched.length end |
#terminate ⇒ Object
Set the scan pointer to the end of the string and clear matching data
153 154 155 156 |
# File 'vendor/bundler/ruby/3.3.0/bundler/gems/ebnf-c8f40958c6c3/lib/ebnf/ll1/scanner.rb', line 153 def terminate feed_me super end |
#unscan ⇒ Object
Sets the scan pointer to the previous position. Only one previous position is remembered, and it changes with each scanning operation.
146 147 148 149 |
# File 'vendor/bundler/ruby/3.3.0/bundler/gems/ebnf-c8f40958c6c3/lib/ebnf/ll1/scanner.rb', line 146 def unscan @lineno = @previous_lineno super end |