Database Management
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Dev Shed ForumsDatabasesDatabase Management

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old December 12th, 2003, 07:32 PM
persaltier's Avatar
persaltier persaltier is offline
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Aug 2001
Location: NYC
Posts: 57 persaltier User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 32 m 19 sec
Reputation Power: 8
Send a message via AIM to persaltier
DB design - flat files or in a table

ok, first an explanation of the project:

the client is a utilities company, and they will be providing the application with CSV files that correlate to usage data for a group of meters over a certain time period. the CSV file will have 2 dates, a start date and an end date .

here's the tricky part.
CSV file #1 can have a start date of Feb 1 and an end date of Feb 30
CSV file #2 can have a start date of Feb 1 and an end date of Feb 15
CSV file #3 can have a start date of Feb 1 and an end date of Feb 30 (same as CSV file #1), except certain data can be different for specific meters in that file.

So simply importing the CSV files into a table by defining the starting and end dates won't work b/c of multiple data sets for the same bounding dates.

Options i've considered:
1. Creating another column in the table that is a csv_file_id. In other words, each file that's uploaded to the application gets assigned an id in another table, and this foreign key is also inputted into the meters table. So searches in the meters table will be done based on csv_file_id , not start / end dates.

2. Create a table for each CSV file. While this will create more tables for MySQL to handle, in a 12-month period, there will be no more than 50 or so tables generated from the CSV files. This will dramatically cut down on the query time as more CSV files are entered, as each CSV file can have upwards of 1,500 lines.

3. Only generate a temporary table for each CSV file at a time when it is called by the application to perform queries on it. This utilizes the same smaller-sized table as in #2, but the idea of generating a temporary table each time a query is called could become too taxing as the # of users of the application increases (the application will handle possibly 25 simultaneous users in the beginning, but that # may grow into the hundreds).

I'm leaning towards #2, as 50 tables isn't that many for MySQL to handle, and the query time to pick out the required CSV file won't be necessary.

Thoughts?

Reply With Quote
  #2  
Old December 12th, 2003, 08:39 PM
r937's Avatar
r937 r937 is offline
SQL Consultant
Click here for more information.
 
Join Date: Feb 2003
Location: Toronto Canada
Posts: 17,982 r937 User rank is General 1st Grade (Above 100000 Reputation Level)r937 User rank is General 1st Grade (Above 100000 Reputation Level)r937 User rank is General 1st Grade (Above 100000 Reputation Level)r937 User rank is General 1st Grade (Above 100000 Reputation Level)r937 User rank is General 1st Grade (Above 100000 Reputation Level)r937 User rank is General 1st Grade (Above 100000 Reputation Level)r937 User rank is General 1st Grade (Above 100000 Reputation Level)r937 User rank is General 1st Grade (Above 100000 Reputation Level)r937 User rank is General 1st Grade (Above 100000 Reputation Level)r937 User rank is General 1st Grade (Above 100000 Reputation Level)r937 User rank is General 1st Grade (Above 100000 Reputation Level)r937 User rank is General 1st Grade (Above 100000 Reputation Level)r937 User rank is General 1st Grade (Above 100000 Reputation Level)r937 User rank is General 1st Grade (Above 100000 Reputation Level)r937 User rank is General 1st Grade (Above 100000 Reputation Level)r937 User rank is General 1st Grade (Above 100000 Reputation Level) 
Time spent in forums: 1 Month 4 Weeks 18 h 20 m 26 sec
Reputation Power: 1028
i like #1

50 files of 1500 lines each is a trivially small table -- okay, medium-sized -- and with proper indexes, all the queries you can think of to run against it will perform fine

dramatically fine

rudy

Reply With Quote
Reply

Viewing: Dev Shed ForumsDatabasesDatabase Management > DB design - flat files or in a table


Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 2 hosted by Hostway
Stay green...Green IT