Hi,
Oleg Bartunov wrote:
>>> if you spidering via http you could ask if page was modified
>>> since last rebuild
>>> and you'll save a lot. Check is your servers actually
>>> provides such information,
>>> probably you need to configure them properly.
>>
>>
>> Most of the pages are dynamically built by php code from numerous files
>> & the database, so the server isn't going to know if they have changed
>> :-(
>
>
> So, that php-code should be corrected to issue right headers. Marc should
> know how.
Please note that code to issue right headers will not necessarily work
faster than the code to just dispatch the page.