PostgreSQL Help
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me

The Shed is going Social! Join us on FaceBook and Twitter and chime in on the conversation.

Go Back   Dev Shed ForumsDatabasesPostgreSQL Help

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old September 28th, 2012, 08:45 AM
antthelimey antthelimey is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Sep 2012
Posts: 1 antthelimey User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 3 m 34 sec
Reputation Power: 0
"Select * " on 12-18M row table from remote machine thru JDBC - Performance nose-div

On machine 1 - a table that contains between 12 and 18 million rows
On machine 2 - a Java app that calls Select * on the table, and writes it into a Lucene index

Originally had a fetchSize of 10,000 and would take around 38 minutes for 12 million, 50 minutes for 16ish million to read it all & write it all back out as the lucene index

One day it started taking 4 hours. If something changed, we dont know what it was

We tracked it down to, after 10 million or so rows, the Fetch to get the next 10,000 rows from the DB goes from like 1 second to 30 seconds, and stays there

After spending a week of two devs & DBA trying to solve this, we eventually "solved" it by upping the FetchRowSize in the JDBC call to 50,000

It was performing well enough again for a few weeks

then...one day... it started taking 4 hours again

we tried upping the shared_buffer from 16GB to 20GB

And last night... it took 7 hours

does anyone have ANY ideas?!

thanks much

Reply With Quote
Reply

Viewing: Dev Shed ForumsDatabasesPostgreSQL Help > "Select * " on 12-18M row table from remote machine thru JDBC - Performance nose-div

Developer Shed Advertisers and Affiliates



Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump

Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 


Powered by: vBulletin Version 3.0.5
Copyright ©2000 - 2013, Jelsoft Enterprises Ltd.

© 2003-2013 by Developer Shed. All rights reserved. DS Cluster - Follow our Sitemap