Re: bytea memory improvement
От | Luis Vilar Flores |
---|---|
Тема | Re: bytea memory improvement |
Дата | |
Msg-id | 44A5B2C3.5040506@evolute.pt обсуждение исходный текст |
Ответ на | Re: bytea memory improvement (Kris Jurka <books@ejurka.com>) |
Список | pgsql-jdbc |
Kris Jurka wrote: > > > On Wed, 28 Jun 2006, Luis Vilar Flores wrote: > >> Can any JDBC developer give some feedback about this patch ? >> Memory usage is a real issue for a few applications ... >> > > It looks like a reasonable thing to do, but could you give us some > more details on the cost/benefits? My sugestion is to trade some CPU time for memory in large files. The benefits are: memory saving (one byte[] of final size at least), less garbage collecting, these buffers are always created on a bytea read, we also save the CPU time of one System.arraycopy, but this is a native method and should be very fast, so it's not a big deal. The only cost is one more pass through the original buffer, this pass is just to count the final array size. In simple terms, the original method did: incoming buffer size N allocate temp buffer size N for through incoming buffer, translate data from incoming to temp allocate final buffer, size M (M is between N and N/4) copy temp buffer to final buffer return final buffer The new method does: incoming buffer size N for through incoming buffer, calculate final buffer size allocate final buffer, size M (M is between N and N/4) for through incoming buffer, translate data from incoming to final return final buffer > Your original email said it cost an extra 30ms for 5MB of data on > your machine. What's percentage > of the original cost is this? This cost is to read and count the escaped bytes of a 5MB byte array in a for cycle on a CeleronM 1.6, but keep in mind that we save the last System.arraycopy (last if in the method). Originally this extra passage didn't existed. I will make more detailed timing in the full method body (old and new), and send results tomorrow. > Also you could be more clear on what percentage of memory this saves. > For the worst case scenario your going to get four bytes of escaped > data for every real byte so the total size of the original method > would be 4 + 4 + 1 and for the new method 4 + 1, so a savings of 44%? > Is that what you've calculated? Yeap, that's for files larger than 1MB on the incoming buffer (the threshold could be defined by some property) - with the extra pass on the incoming array I calculate the final size, so I can skip the temp buffer, so we save the 44% (on worst case), the minimum savings is 16.7% for no escaped data. > > Finally you haven't actually submitted a patch, you've just sent a > modified copy of a whole file. Since it's a small file that changes > infrequently it's not a big deal, but we prefer context diffs if you can. > > Kris Jurka > > ---------------------------(end of broadcast)--------------------------- > TIP 1: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly > I will send the diff too. Thanks for the comments ... -- Luis Flores Analista de Sistemas *Evolute* - Consultoria Informática <http://www.evolute.pt> Email: lflores@evolute.pt <mailto:lflores@evolute.pt> Tel: (+351) 212949689 AVISO DE CONFIDENCIALIDADE Esta mensagem de correio electrónico e eventuais ficheiros anexos são confidenciais e destinados apenas à(s) pessoa(s) ou entidade(s) acima referida(s), podendo conter informação privilegiada e confidencial, a qual não poderá ser divulgada, copiada, gravada ou distribuída nos termos da lei vigente. Caso não seja o destinatário da mensagem, ou se ela lhe foi enviada por engano, agradecemos que não faça uso ou divulgação da mesma. A distribuição ou utilização da informação nela contida é interdita. Se recebeu esta mensagem por engano, por favor notifique o remetente e apague este e-mail do seu sistema. Obrigado. CONFIDENTIALITY NOTICE This e-mail transmission and eventual attached files are intended only for the use of the individual(s) or entity(ies) named above and may contain information that is both privileged and confidential and is exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in this transmission is strictly restricted. If by any means you have received this transmission in error, please immediately notify the sender and delete this e-mail from your system. Thank you.
В списке pgsql-jdbc по дате отправления: