Breaking text into matching chunks

Hi all,

I'm trying to break up a string into quoted strings, commented strings, brackets (just the character) and other strings. It happens to be SQL, but it's not SQL specific.

For example, if I have:

create view "My View Name" -- this is where I define the name
as select
myFirstColumn || (select me from mine)
, ' a static literal' as mySecondColumn /* weird */

Then I want to break it up into successive chunks:

1: create view
2: "My View Name"
3: -- this is where I define the name\n
4: as select\n myFirstColumn ||
5: (
6: select me from mine
7: )\n ,
8: ' a static literal'
6: as mySecondColumn
9: /* weird */

I tried this regex:

m/(?xism)
( # one of the following
".*?"|'.*?'|\[.*?\] # quoted
| --.*?(?:\r\n|\n|\r)|/[*].*?[*]/ # comments
| \(|\) # brackets
| .+? # other
)
/g

but it's not quite right. Any ideas?

Thanks,
Tom
BareFeet

Having had time to think about it, I don't think I'd use a regular expression for this.

What programming language are you working with?

I agree, I started working on this Friday and didn't get anywhere. He appears to be using Perl. I would walk through the string one character at a time and set yourself flags for things like comments and quoted strings.

-Dan

Maybe you could use a regex. Multiple, actually.

One generic will read a string until it reaches a delimiter: a quote, --, (, ), or /*.

Depending on that second group, you use another regex for the next part:

( doesn't have one (it's just the character) and ) uses the generic.

I managed to do it by just modifying my regular expression to include a lookahead after the "other" grouping for any known groups. It basically says "find on of the following up to just before the next occurrence of another match)

m/(?xis)
( # one of the following
".*?"|'.*?'|\[.*?\] # quoted
| --.*?(?:\r\n|\n|\r) # comments at end of line
| /[*].*?[*]/ # comments in line
| \(|\) # brackets
| .+? # other
(?= # look ahead for one of the following
".*?"|'.*?'|\[.*?\] # quoted
| --.*?(?:\r\n|\n|\r) # comments at end of line
| /[*].*?[*]/ # comments in line
| \(|\) # brackets
)
)/g

It gives me the result I needed:

1: create view
2: "My View Name"
3: <space>
4: -- this is where I define the name\n
5: as select\n myFirstColumn ||
6: (
7: select me from mine
8: )
9: \n ,
10: ' a static literal'
11: as mySecondColumn
12: /* weird */

Obviously this RegEx is scanning some of the text twice, which is slightly inefficient. I thought that the /g option would be smart enough to interpret a .+? within the RegEx as meaning up to just before the next iteration of the grouping, but I guess not, so I have to include the lookahead to do this specifically.

And of course I can simplify the lookahead to only look for the starting characters of the quotes and comments, so the RegEx is:

m/(?xis)
( # one of the following
".*?"|'.*?'|\[.*?\] # quoted
| --.*?(?:\r\n|\n|\r) # comments at end of line
| /[*].*?[*]/ # comments in line
| \(|\) # brackets
| .+? # other
(?= # look ahead for one of the following
"|'|\[ # quoted
| -- # comments at end of line
| /[*] # comments in line
| \(|\) # brackets
)
)/g



I'm using AppleScript, currently calling perl to do the actual regex, but may move that function into a scripting addition that does regex, or even some Cocoa subroutine. So I'm after a generic PCRE RegEx that I can use in whatever environment.

Thanks,
Tom
BareFeet