Other Programming Languages
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Dev Shed ForumsProgramming Languages - MoreOther Programming Languages
The ASP Free website provides in-depth information on the latest developer tools available from Microsoft. Our cadre of writers, highly experienced industry experts, reveals the best ways to use established technologies as well as new and emerging technologies. Our coverage of Microsoft's development and administration technologies is among the most respected in the IT industry today.

ASP Free and Iron Speed Designer are giving away $5,500+ in FREE licenses. Iron Speed's RAD CASE toolset can save up to 80% of your coding time. One free license per week, one perpetual license per month!
Download and Activate to enter!

Intel® Graphics Performance Analyzers is a powerful tool suite for analyzing and optimizing your games, media, and graphics-intensive applications. Used by some of the best developers on the planet, Intel GPA lets you maximize your app’s performance.


Tutorials
| Forums

Download to Enter
| Contest Rules

DOWNLOAD INTEL® GPA FOR FREE

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Rating: Thread Rating: 3 votes, 4.33 average. Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old December 16th, 2005, 07:07 PM
Scorpions4ever's Avatar
Scorpions4ever Scorpions4ever is offline
Banned ;)
Dev Shed God 8th Plane (8500 - 8999 posts)
 
Join Date: Nov 2001
Location: Woodland Hills, Los Angeles County, California, USA
Posts: 8,978 Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level) 
Time spent in forums: 1 Month 2 Weeks 6 Days 21 h 44 m 27 sec
Reputation Power: 3561
How does an interpreter/compiler work?

Everything you were afraid to ask about how a compiler/interpreter works.

Let's start by discussing a very simple form of a compiler/interpreter:

Source File ---> Scanner ---> Lexer ---> Parser ---> Interpreter/Code Generator

So what do all the terms here mean. We'll work on that as we go:
Source File: This is the program that is read by the compiler or interpreter. This is the text that needs to be compiled or interpreted.

Scanner: This is the first module in a compiler or interpreter. Its job is to read the source file one character at a time. It can also keep track of which line number and character is currently being read. A typical scanner can be instructed to move backwards and forwards through the source file. Why do we need to move backwards? We will see why in just a little bit when we examine the lexer. For now, assume that each time the scanner is called, it returns the next character in the file.

Lexer: This module serves to break up the source file into chunks (called tokens). It calls the scanner to get characters one at a time and organizes them into tokens and token types. For instance, if the source file read something like this:
Code:
cx = cy + 324;
print "value of cx is ", cx;

a lexer would perhaps break it like this:
Code:
cx  --> Identifier (variable)
=  --> Symbol (assignment operator)
cy  --> Identifier (variable)
+ --> Symbol (addition operator)
324 --> Numeric constant (integer)
; --> Symbol (end of statement)
print --> Identifier (keyword)
"value of cx is " --> String constant
, --> Symbol (string concatenation operator)
cx --> Identifier (variable)
; --> Symbol (end of statement)

Thus, the lexer calls the scanner to pass it one character at a time and groups them together and identifies them up as tokens for the language parser (which is the next stage). It also identifies the type of token (variable vs. keyword, assignment operator vs. addition operator vs. string concatenation operator etc.) Occasionally, the lexer has to tell the scanner to back up though. Consider a language that has operators that may be more than one character long (! vs. !=, < vs. <=, + vs. ++ etc.). Assume that the lexer has requested the scanner for a character and it has returned '<'. The lexer needs to determine whether the operator is a < or a <=. So it requests the scanner for another character. If the next character is a '=', it changes the token to '<=' and passes it to the parser. If not, it tells the scanner to back up one character and hold it in the buffer, while it passes the '<' to the parser.

Parser: This is the part of the compiler that really understands the syntax of the language. It calls the lexer to get tokens and processes the tokens per the syntax of the language. For instance, taking the example from the lexer above, the hypothetical interaction between the lexer and parser could go like this:
Code:
Parser: Give me the next token
Lexer: Next token is "cx" which is a variable.
Parser: Ok, I have "cx" as a declared integer variable. Give me next token
Lexer: Next token is "=", the assignment operator.
Parser: Ok, the program wants me to assign something to "cx". Next token 
     Lexer: The next token is "cy" which is a variable.
     Parser: Ok, I know "cy" is an integer variable. Next token please
     Lexer: The next token is '+', which is an addition operator.
     Parser: Ok, so I need to add something to the value in "cy". Next token please.
         Lexer: The next token is "324", which is an integer.
         Parser: Ok, both "cy" and "324" are integers, so I can add them. Next token please:
         Lexer: The next token is ";" which is end of statement.
     Parser: Ok, I will evaluate "cy + 324" and get the answer
Parser: I'll take the answer from "cy + 324" and assign it to "cx"

In the above, the indenting shows a subprocess that the parser enters, to evaluate "cy + 324". This gives you a decent idea about how the parser operates. Also note that the parser is checking types and syntax rules (for instance, it checked whether cy and 324 were both integer types before adding them). If the parser gets a token that it was not expecting, it will stop processing and complain to the user about an error. The Scanner holds the current line number and character, so the Parser can inform the user approximately where the error occurred.

Interpreter/Code Generator: This is the part that actually takes the action that is specified by a program statement. In some cases, this is actually part of the parser (especially for interpreters) and the parser interprets and takes action directly. In other cases, the parser converts the statements into byte-code (intermediate language). In case of a compiler, it then hands them to the Code Generator to convert into machine code instructions. If you want a compiler for a different CPU or architecture, all you have to do is put a new code generator unit to translate the byte code into machine code for the new CPU.

This is about the simplest form for an interpreter or compiler. In the next few threads, we will look in some more detail at the interaction between the Parser and Code Generator.

Feedback about this post will be greatly appreciated.
Comments on this post
SimonGreenhill agrees: informative
medialint agrees!
netytan agrees: Awesum Scorpi
jafet agrees: Good article
JhonnyO@gmail.c agrees!
crownjewel82 agrees: My compiler construction professor would be proud.
LinuxPenguin agrees: Amazing
Brokenhope agrees: Thank you. I have always been curious how a compiler worked.
__________________
Up the Irons
What Would Jimi Do? Smash amps. Burn guitar. Take the groupies home.
"Death Before Dishonour, my Friends!!" - Bruce D ickinson, Iron Maiden Aug 20, 2005 @ OzzFest
Down with Sharon Osbourne

Diary of a first time dog owner <-- my cousin's blog

Last edited by Scorpions4ever : December 16th, 2005 at 07:52 PM.

Reply With Quote
  #2  
Old November 6th, 2008, 03:53 AM
duabevnh duabevnh is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Nov 2008
Posts: 3 duabevnh Negative: is most likely a SPAMMER and a traitor to the cause. 
Time spent in forums: 22 m 34 sec
Reputation Power: 0
Thanks you!
Have you source?
Comments on this post
MrFujin disagrees: can you think? sorry, but answering a 3 years old post with a question that cant be answered is not
that great (source in what!?)
FenderStrat agrees!

Reply With Quote
  #3  
Old November 6th, 2008, 01:28 PM
Scorpions4ever's Avatar
Scorpions4ever Scorpions4ever is offline
Banned ;)
Dev Shed God 8th Plane (8500 - 8999 posts)
 
Join Date: Nov 2001
Location: Woodland Hills, Los Angeles County, California, USA
Posts: 8,978 Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level) 
Time spent in forums: 1 Month 2 Weeks 6 Days 21 h 44 m 27 sec
Reputation Power: 3561
Actually I do have source code . People are using it as well.

Reply With Quote
  #4  
Old November 7th, 2008, 08:37 AM
duabevnh duabevnh is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Nov 2008
Posts: 3 duabevnh Negative: is most likely a SPAMMER and a traitor to the cause. 
Time spent in forums: 22 m 34 sec
Reputation Power: 0
Can you share?
Thanks a lot!

Reply With Quote
  #5  
Old November 8th, 2008, 04:11 PM
Scorpions4ever's Avatar
Scorpions4ever Scorpions4ever is offline
Banned ;)
Dev Shed God 8th Plane (8500 - 8999 posts)
 
Join Date: Nov 2001
Location: Woodland Hills, Los Angeles County, California, USA
Posts: 8,978 Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level)Scorpions4ever User rank is General 39th Grade (Above 100000 Reputation Level) 
Time spent in forums: 1 Month 2 Weeks 6 Days 21 h 44 m 27 sec
Reputation Power: 3561
What do you need it for? Just curious?

Reply With Quote
  #6  
Old November 13th, 2008, 03:36 PM
duabevnh duabevnh is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Nov 2008
Posts: 3 duabevnh Negative: is most likely a SPAMMER and a traitor to the cause. 
Time spent in forums: 22 m 34 sec
Reputation Power: 0
I learn compiler.I'm programming Scanner, but I have problem!

Reply With Quote
  #7  
Old October 1st, 2009, 12:54 AM
Nyan Nyan is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Jan 2009
Posts: 337 Nyan User rank is Sergeant Major (2000 - 5000 Reputation Level)Nyan User rank is Sergeant Major (2000 - 5000 Reputation Level)Nyan User rank is Sergeant Major (2000 - 5000 Reputation Level)Nyan User rank is Sergeant Major (2000 - 5000 Reputation Level)Nyan User rank is Sergeant Major (2000 - 5000 Reputation Level)Nyan User rank is Sergeant Major (2000 - 5000 Reputation Level) 
Time spent in forums: 5 Days 2 h 15 m 47 sec
Reputation Power: 44
Hey, I'm writing a small interpreter(inspired by High Order Calculator by Kernighan & Rob Pike) using yacc and hand-written lexical analyser. Currently I'm using function pointers as like in HOC.
which would be better switch/function pointer ?
Where should I store my function parameters? because for recursive call with formal parameters I need to restore parameters. What's the most efficient way?

Reply With Quote
  #8  
Old January 11th, 2010, 09:18 AM
dmainyeo dmainyeo is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Jan 2010
Posts: 1 dmainyeo User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 5 m 28 sec
Reputation Power: 0
Target code generator

Hi can someone help me with source code for a target code generator. Its for my assignment due friday

Reply With Quote
Reply

Viewing: Dev Shed ForumsProgramming Languages - MoreOther Programming Languages > How does an interpreter/compiler work?


Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump

Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 


Powered by: vBulletin Version 3.0.5
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.

© 2003-2012 by Developer Shed. All rights reserved. DS Cluster 10 - Follow our Sitemap