Thread: CSV Parsing

    #1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2013
    Posts
    5
    Rep Power
    0

    CSV Parsing


    Hi.

    I'm trying to parse a .csv file which contains the following data

    2, 0, Pitch_bend_c, 0, 8242
    2, 0, Control_c, 0, 11, 38
    2, 0, Note_on_c, 0, 36, 71

    What i would like to do is convert some events like notes and duration to a new csv but my main problem is searching through the csv file for specific numbers to apply the conversion.

    Here's what i mean, given this line
    2, 0, Note_on_c, 0, 36, 71

    Get the 36 and convert it to C2, get the 37 and convert it to C# and so on.

    If somebody knows of a sample to help me it would be appreciated.
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Intermediate (1500 - 1999 posts)

    Join Date
    Apr 2009
    Posts
    1,875
    Rep Power
    1225
    The line you reference doesn't contain a 37 in any of the fields so no conversion.

    What are the conversion rules? Why would 36 be converted to C2 and 37 be converted to C#?

    What does your current code look like and how does it fail to do what you want?
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2013
    Posts
    5
    Rep Power
    0
    The 37 was another example that could could occur on the next line.

    The conversion rules would be based on the MIDI CC notes table that i want to incorporate on the script.

    I still haven't got any code written just a sample of a transposition script that looks like this

    Code:
        $offset = -12;
        $percussion = 9;
    
        while ($a = <>) {
    
            #   Recognise Note_on_c and Note_off_c records and crack into:
    
            #       $1  Start of record
            #       $2  Channel number
            #       $3  Note number
            #       $a  Balance of record
    
            if ($a =~ s/(\d+,\s*\d+,\s*Note_\w+,\s*(\d+),\s*)(\d+)//) {
                $n = $3;
                if ($2 != $percussion) {
                    $n += $offset;
                }
                if ($n < 0) {
                    next;
                }
                $a = "$1$n$a";
            }
            print($a);
        }
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Intermediate (1500 - 1999 posts)

    Join Date
    Apr 2009
    Posts
    1,875
    Rep Power
    1225
    Since you haven't included any info on the "MIDI CC notes table", I can't help you with coding the conversion.

    Don't use a regex for parsing csv data. Use the Text::CSV module.

    $a is one of perl's built-in global vars used in sort routines (i.e., the sort function). It is best not to use it outside of the sort routine.

    Your script should start out with these 2 pragmas:
    Code:
    use strict;
    use warnings;
    The strict pragma will require you to declare your vars, which is done with the 'my' keyword.
    Code:
        my $offset = -12;
        my $percussion = 9;
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2013
    Posts
    5
    Rep Power
    0
    Where is the MIDI CC table.

    Code:
    http://www.tonalsoft.com/pub/news/pitch-bend.aspx
    The conversion would be from the MIDI note number to the note name and frequency.

    Thanks for the help so far.
  10. #6
  11. No Profile Picture
    Contributing User
    Devshed Intermediate (1500 - 1999 posts)

    Join Date
    Apr 2009
    Posts
    1,875
    Rep Power
    1225
    Create a hash where the keys are the MIDI note numbers and the values are the corresponding note names.
    Example:
    Code:
    my %num2note = (
        0  => 'C',
        1  => 'C#',
        2  => 'D',
        # etc
    );
    As you parse each line in the csv file (using Text::CSV), you simply lookup the note number in the hash and replace that field with the corresponding note name.

    Once each required field in the row has been updated, you output that updated row data to the new csv file.
    Last edited by FishMonger; July 28th, 2013 at 12:14 PM.
  12. #7
  13. No Profile Picture
    Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Jun 2012
    Posts
    776
    Rep Power
    495
    Once you have done the hash for the first octave (12 semitones), use a loop to fill it for the next octaves.
  14. #8
  15. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2013
    Posts
    5
    Rep Power
    0
    So i've started to write something.

    Here's what i have so far

    Code:
    #!/usr/bin/perl
    use strict;
    use warnings;
    use Text::CSV;
    
    my %num2note = (
    	0 => 'C-5',
    	1 => 'C#-5',
    	2 => 'D-5',
    	3 => 'D#-5',
    	4 => 'E-5',
    	5 => 'F-5',
    	6 => 'F#-5',
    	7 => 'G-5',
    	8 => 'G#-5',
    	9 => 'A-5',
    	10 => 'A#-5',
    	11 => 'B-5',
    	12 => 'C-4',
    	13 => 'C#-4',
    	14 => 'D-4',
    	15 => 'D#-4',
    	16 => 'E-4',
    	17 => 'F-4',
    	18 => 'F#-4',
    	19 => 'G-4',
    	20 => 'G#-4',
    	21 => 'A-4',
    	22 => 'A#-4',
    	23 => 'B-4',
    	24 => 'C-3',
    	25 => 'C#-3',
    	26 => 'D-3',
    	27 => 'D#-3',
    	28 => 'E-3',
    	29 => 'F-3',
    	30 => 'F#-3',
    	31 => 'G-3',
    	32 => 'G#-3',
    	33 => 'A-3',
    	34 => 'A#-3',
    	35 => 'B-3',
    	36 => 'C-2',
    	37 => 'C#-2',
    	38 => 'D-2',
    	39 => 'D#-2',
    	40 => 'E-2',
    	41 => 'F-2',
    	42 => 'F#-2',
    	43 => 'G-2',
    	44 => 'G#-2',
    	45 => 'A-2',
    	46 => 'A#-2',
    	47 => 'B-2',
    	48 => 'C-1',
    	49 => 'C#-1',
    	50 => 'D-1',
    	51 => 'D#-1',
    	52 => 'E-1',
    	53 => 'F-1',
    	54 => 'F#-1',
    	55 => 'G-1',
    	56 => 'G#-1',
    	57 => 'A-1',
    	58 => 'A#-1',
    	59 => 'B-1',
    	60 => 'C0',
    	61 => 'C#0',
    	62 => 'D0',
    	63 => 'D#0',
    	64 => 'E0',
    	65 => 'F0',
    	66 => 'F#0',
    	67 => 'G0',
    	68 => 'G#0',
    	69 => 'A0',
    	70 => 'A#0',
    	71 => 'B0',
    	72 => 'C1',
    	73 => 'C#1',
    	74 => 'D1',
    	75 => 'D#1',
    	76 => 'E1',
    	77 => 'F1',
    	78 => 'F#1',
    	79 => 'G1',
    	80 => 'G#1',
    	81 => 'A1',
    	82 => 'A#1',
    	83 => 'B1',
    	84 => 'C2',
    	85 => 'C#2',
    	86 => 'D2',
    	87 => 'D#2',
    	88 => 'E2',
    	89 => 'F2',
    	90 => 'F#2',
    	91 => 'G2',
    	92 => 'G#2',
    	93 => 'A2',
    	94 => 'A#2',
    	95 => 'B2',
    	96 => 'C3',
    	97 => 'C#3',
    	98 => 'D3',
    	99 => 'D#3',
    	100 => 'E3',
    	101 => 'F3',
    	102 => 'F#3',
    	103 => 'G3',
    	104 => 'G#3',
    	105 => 'A3',
    	106 => 'A#3',
    	107 => 'B3',
    	108 => 'C4',
    	109 => 'C#4',
    	110 => 'D4',
    	111 => 'D#4',
    	112 => 'E4',
    	113 => 'F4',
    	114 => 'F#4',
    	115 => 'G4',
    	116 => 'G#4',
    	117 => 'A4',
    	118 => 'A#4',
    	119 => 'B4',
    	120 => 'C5',
    	121 => 'C#5',
    	122 => 'D5',
    	123 => 'D#5',
    	124 => 'E5',
    	125 => 'F5',
    	126 => 'F#5',
    	127 => 'G5',
    	);
    
    my $file = 'out.csv';
    
    my $csv = Text::CSV->new();
    
    open (CSV, "<", $file) or die $!;
    
    while (<CSV>) {
            next if ($. == 1);
            if ($csv->parse($_)) {
                my @columns = $csv->fields();
                print "Track: $columns[0]\nTime: $columns[1]\nType: $columns[2]\nChannel: $columns[3]\nNote: $columns[4]\nVelocity: $columns[5]\n";
            } else {
                my $err = $csv->error_input;
                print "Failed to parse line: $err";
            }
        }
    close CSV;
    My problem now is to print the correct columns on each line and match them with the note names.

    Here's another example:

    2, 110, Pitch_bend_c, 0, 7986 <- this line has a only 5 rows
    2, 0, Note_on_c, 0, 36, 71 <- this has 6 rows

    All parameters are different for each line so i want to for the first line:

    Track : 2, Time: 110, Type: Pitch_bend_c, Channel: 0, Value: 7986

    and the second one

    Track: 2, Time: 0, Type: Note_on_c, Channel: 0, Note: C-2, Velocity: 71.

    Sorry for all these questions, i'm new to this.
  16. #9
  17. No Profile Picture
    Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Jun 2012
    Posts
    776
    Rep Power
    495
    You should probably use the Text::CSV module, but I'll give you a simple split/regex solution:

    Perl Code:
    my ($track, $time, $type, $channel, @rest) = split /[\s,]+/, $line;
    if (@rest == 2) {
         my ($note, $velocity) = @rest;
         $note = $num2note{$note};
    } else {
         $value = shift @rest;
    }
  18. #10
  19. No Profile Picture
    Contributing User
    Devshed Intermediate (1500 - 1999 posts)

    Join Date
    Apr 2009
    Posts
    1,875
    Rep Power
    1225
    In a properly formatted csv file, each row should have the same number of fields. Some of those fields could be empty.

    As I understand it, you only want to modify the rows that contain "Note" in the 3rd field. Is that correct? Will that row always have the same number of fields and is the 5th field the only one that needs to be altered?

    Do you want all lines in the new csv file, or just the ones that were updated?
  20. #11
  21. No Profile Picture
    Contributing User
    Devshed Intermediate (1500 - 1999 posts)

    Join Date
    Apr 2009
    Posts
    1,875
    Rep Power
    1225
    Code:
    #!/usr/bin/perl
    
    use strict;
    use warnings;
    use Text::CSV;
    
    my %num2note = (
    	0 => 'C-5',
    	1 => 'C#-5',
    	2 => 'D-5',
    	3 => 'D#-5',
    	4 => 'E-5',
    	5 => 'F-5',
    	6 => 'F#-5',
    	7 => 'G-5',
    	8 => 'G#-5',
    	9 => 'A-5',
    	10 => 'A#-5',
    	11 => 'B-5',
    	12 => 'C-4',
    	13 => 'C#-4',
    	14 => 'D-4',
    	15 => 'D#-4',
    	16 => 'E-4',
    	17 => 'F-4',
    	18 => 'F#-4',
    	19 => 'G-4',
    	20 => 'G#-4',
    	21 => 'A-4',
    	22 => 'A#-4',
    	23 => 'B-4',
    	24 => 'C-3',
    	25 => 'C#-3',
    	26 => 'D-3',
    	27 => 'D#-3',
    	28 => 'E-3',
    	29 => 'F-3',
    	30 => 'F#-3',
    	31 => 'G-3',
    	32 => 'G#-3',
    	33 => 'A-3',
    	34 => 'A#-3',
    	35 => 'B-3',
    	36 => 'C-2',
    	37 => 'C#-2',
    	38 => 'D-2',
    	39 => 'D#-2',
    	40 => 'E-2',
    	41 => 'F-2',
    	42 => 'F#-2',
    	43 => 'G-2',
    	44 => 'G#-2',
    	45 => 'A-2',
    	46 => 'A#-2',
    	47 => 'B-2',
    	48 => 'C-1',
    	49 => 'C#-1',
    	50 => 'D-1',
    	51 => 'D#-1',
    	52 => 'E-1',
    	53 => 'F-1',
    	54 => 'F#-1',
    	55 => 'G-1',
    	56 => 'G#-1',
    	57 => 'A-1',
    	58 => 'A#-1',
    	59 => 'B-1',
    	60 => 'C0',
    	61 => 'C#0',
    	62 => 'D0',
    	63 => 'D#0',
    	64 => 'E0',
    	65 => 'F0',
    	66 => 'F#0',
    	67 => 'G0',
    	68 => 'G#0',
    	69 => 'A0',
    	70 => 'A#0',
    	71 => 'B0',
    	72 => 'C1',
    	73 => 'C#1',
    	74 => 'D1',
    	75 => 'D#1',
    	76 => 'E1',
    	77 => 'F1',
    	78 => 'F#1',
    	79 => 'G1',
    	80 => 'G#1',
    	81 => 'A1',
    	82 => 'A#1',
    	83 => 'B1',
    	84 => 'C2',
    	85 => 'C#2',
    	86 => 'D2',
    	87 => 'D#2',
    	88 => 'E2',
    	89 => 'F2',
    	90 => 'F#2',
    	91 => 'G2',
    	92 => 'G#2',
    	93 => 'A2',
    	94 => 'A#2',
    	95 => 'B2',
    	96 => 'C3',
    	97 => 'C#3',
    	98 => 'D3',
    	99 => 'D#3',
    	100 => 'E3',
    	101 => 'F3',
    	102 => 'F#3',
    	103 => 'G3',
    	104 => 'G#3',
    	105 => 'A3',
    	106 => 'A#3',
    	107 => 'B3',
    	108 => 'C4',
    	109 => 'C#4',
    	110 => 'D4',
    	111 => 'D#4',
    	112 => 'E4',
    	113 => 'F4',
    	114 => 'F#4',
    	115 => 'G4',
    	116 => 'G#4',
    	117 => 'A4',
    	118 => 'A#4',
    	119 => 'B4',
    	120 => 'C5',
    	121 => 'C#5',
    	122 => 'D5',
    	123 => 'D#5',
    	124 => 'E5',
    	125 => 'F5',
    	126 => 'F#5',
    	127 => 'G5',
    	);
    
    my $file = 'out.csv';
    
    my $csv = Text::CSV->new( {eol => "\n"} );
    
    #open (CSV, "<", $file) or die $!;
    
    while (<DATA>) {
            #next if ($. == 1);
            if ($csv->parse($_)) {
                my @columns = $csv->fields();
                s/^\s+|\s+$//g for @columns;
                $columns[4] = $num2note{$columns[4]} if $columns[2] =~ /^Note/;
                $csv->print(\*STDOUT, \@columns);
            } else {
                my $err = $csv->error_input;
                print "Failed to parse '$_': $err";
            }
        }
    #close CSV;
    
    __DATA__
    2, 0, Pitch_bend_c, 0, 8242
    2, 0, Control_c, 0, 11, 38
    2, 0, Note_on_c, 0, 36, 71
    Output of D:\test>test.pl
    2,0,Pitch_bend_c,0,8242
    2,0,Control_c,0,11,38
    2,0,Note_on_c,0,C-2,71
  22. #12
  23. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2013
    Posts
    5
    Rep Power
    0
    Thank you.

    Using your code i managed to filter some more things, now i'm just trying tweak it a bit and wrap it up.

    Here's what i have

    Code:
    #!/usr/bin/perl
    use strict;
    use warnings;
    use Text::CSV;
    
    my %num2note = (
    	0 => 'C-5',
    	1 => 'C#-5',
    	2 => 'D-5',
    	3 => 'D#-5',
    	4 => 'E-5',
    	5 => 'F-5',
    	6 => 'F#-5',
    	7 => 'G-5',
    	8 => 'G#-5',
    	9 => 'A-5',
    	10 => 'A#-5',
    	11 => 'B-5',
    	12 => 'C-4',
    	13 => 'C#-4',
    	14 => 'D-4',
    	15 => 'D#-4',
    	16 => 'E-4',
    	17 => 'F-4',
    	18 => 'F#-4',
    	19 => 'G-4',
    	20 => 'G#-4',
    	21 => 'A-4',
    	22 => 'A#-4',
    	23 => 'B-4',
    	24 => 'C-3',
    	25 => 'C#-3',
    	26 => 'D-3',
    	27 => 'D#-3',
    	28 => 'E-3',
    	29 => 'F-3',
    	30 => 'F#-3',
    	31 => 'G-3',
    	32 => 'G#-3',
    	33 => 'A-3',
    	34 => 'A#-3',
    	35 => 'B-3',
    	36 => 'C-2',
    	37 => 'C#-2',
    	38 => 'D-2',
    	39 => 'D#-2',
    	40 => 'E-2',
    	41 => 'F-2',
    	42 => 'F#-2',
    	43 => 'G-2',
    	44 => 'G#-2',
    	45 => 'A-2',
    	46 => 'A#-2',
    	47 => 'B-2',
    	48 => 'C-1',
    	49 => 'C#-1',
    	50 => 'D-1',
    	51 => 'D#-1',
    	52 => 'E-1',
    	53 => 'F-1',
    	54 => 'F#-1',
    	55 => 'G-1',
    	56 => 'G#-1',
    	57 => 'A-1',
    	58 => 'A#-1',
    	59 => 'B-1',
    	60 => 'C0',
    	61 => 'C#0',
    	62 => 'D0',
    	63 => 'D#0',
    	64 => 'E0',
    	65 => 'F0',
    	66 => 'F#0',
    	67 => 'G0',
    	68 => 'G#0',
    	69 => 'A0',
    	70 => 'A#0',
    	71 => 'B0',
    	72 => 'C1',
    	73 => 'C#1',
    	74 => 'D1',
    	75 => 'D#1',
    	76 => 'E1',
    	77 => 'F1',
    	78 => 'F#1',
    	79 => 'G1',
    	80 => 'G#1',
    	81 => 'A1',
    	82 => 'A#1',
    	83 => 'B1',
    	84 => 'C2',
    	85 => 'C#2',
    	86 => 'D2',
    	87 => 'D#2',
    	88 => 'E2',
    	89 => 'F2',
    	90 => 'F#2',
    	91 => 'G2',
    	92 => 'G#2',
    	93 => 'A2',
    	94 => 'A#2',
    	95 => 'B2',
    	96 => 'C3',
    	97 => 'C#3',
    	98 => 'D3',
    	99 => 'D#3',
    	100 => 'E3',
    	101 => 'F3',
    	102 => 'F#3',
    	103 => 'G3',
    	104 => 'G#3',
    	105 => 'A3',
    	106 => 'A#3',
    	107 => 'B3',
    	108 => 'C4',
    	109 => 'C#4',
    	110 => 'D4',
    	111 => 'D#4',
    	112 => 'E4',
    	113 => 'F4',
    	114 => 'F#4',
    	115 => 'G4',
    	116 => 'G#4',
    	117 => 'A4',
    	118 => 'A#4',
    	119 => 'B4',
    	120 => 'C5',
    	121 => 'C#5',
    	122 => 'D5',
    	123 => 'D#5',
    	124 => 'E5',
    	125 => 'F5',
    	126 => 'F#5',
    	127 => 'G5',
    	);	
    	
    my %num2ctl = (
    	1 => 'Modulation',
    	7 => 'Volume)',
    	10 => 'Pan',
    	11 => 'Expression',
    	64 => 'Sustain'
    	);
    
    my %num2div = (
    	2 => '4',
    	3 => '8',
    	4 => '16',
    	5 => '32');
    	
    my $file = $ARGV[0] or die $!;;
    
    my $csv = Text::CSV->new( {eol => "\n"} );
    
    open (CSV, "<", $file) or die $!;
    
    while (<CSV>) {
            if ($csv->parse($_)) {
                my @columns = $csv->fields();
                s/^\s+|\s+$//g for @columns;
    			#my $freq = 440 * 2 ** (($columns[4]-69)/12);
    			$columns[4] = $num2div{$columns[4]} if $columns[2] =~ /^Time_signature/;
                $columns[4] = $num2note{$columns[4]} if $columns[2] =~ /^Note/;
    			$columns[4] = $num2ctl{$columns[4]} if $columns[2] =~ /^Control/;
    			my $pitchbend = $freq + (($columns[4]-8192)/(4096*12));
    			$columns[4] = $pitchbend if $columns[2] =~ /^Pitch/;
                $csv->print(\*STDOUT, \@columns);
            } else {
                my $err = $csv->error_input;
                print "Failed to parse '$_': $err";
            }
        }
    close CSV;
    My problem is adding the $freq to columns[4] when Note occurs i've tried to use splice but no luck.
    The calculations on the pitchbend column are coming screwed up too.

IMN logo majestic logo threadwatch logo seochat tools logo