Monday, April 23, 2012

"wget" gets blocked Java

Recently I faced an interesting problem. I wanted to crawl some pages from the internet and chose wget as the weapon. But, as I am a lover of Java, I wanted to call it from Java. Though I regretted a lot last hours  not following my initial plan to call it through Perl (or theoretically any scripting language) but now happy finding a hack.

Why regret? It hangs!! Just after downloading a couple of pages it sleeps to never come back alive. Well, digging out revealed, it completely occupied the stdout and the process is stuck. That means you need to flush the stdout to move forward. This can be handled by either stopping the buffering or continuously flushing the buffer!

So better would be to silent wget before it "silent"s your process. Adding the options -nv and -q, no verbose and quiet respectively, make the wget silent and your process does not get blocked.

So  at the end of the day you choose to call wget from Java, dont worry but mind to add the options -q and  -nv in your command!

3 comments: