Digital Humanities Questions & Answers

1. 

   Karin Dalziel
   Member

   I am looking for software that will download a complete working copy of a web page. I want this specifically for a simple neatline map page right now (one which uses a historical map), which I realize is database backed but I am hoping there is something that can recreate a simple site of this sort.

   I'd like to do this in general for a couple reasons: it would allow us to archive a few things (non-neatline) that are using aging technologies I'm afraid will go away soon, and it would allow me to work on someone else's code locally when they ask me for troubleshooting advice. I have used HTTrack in the past, but it seems for more complex websites it's missing big swaths of both CSS and javascript, or is not linking them up properly.

   Software I have tried: HTTrack, sitesucker, and scrapbook firefox plugin, as well as most browser's "download complete web page" option.

   • Tweet this question

   Posted 4 years ago Permalink
2. 

   Patrick Murray-John
   

   It might be a bit heavy and complicated to set up for what you need (at least last I looked at it, which was a couple years ago), Internet Archive uses Heritrix

   Posted 4 years ago Permalink
3. 

   Josh
   Member

   If you are on a Mac, give SiteCrawler a shot: http://lightheadsw.com/sitecrawler/

   I've used it before and have been happy with the results, including capture of CSS files.

   Posted 4 years ago Permalink
4. 

   Stéfan Sinclair
   Admin

   If you're comfortable on the command-line you might want to try wget (that may already be installed on your system, especially in Linux):

   wget -r WEBSITE_URL

   A bit fancier to try mirroring an entire site (as per your original question)

   $ wget --mirror -p --convert-links -P LOCAL_DIRECTORY WEBSITE_URL

   There are a ton of other options for working in the background, setting user_agent, etc. Googling "wget mirror" should provide additional information.

   Posted 4 years ago Permalink
5. 

   Trip Kirkpatrick
   Member

   Extending @Stéfan Sinclair's post:

   There's a low-slope intro to wget over at Programming Historian 2: http://programminghistorian.org/lessons/automated-downloading-with-wget

   Posted 4 years ago Permalink
6. 

   Karin Dalziel
   Member

   Thanks everyone! It was pointed out to me that exactly what I was trying to do isn't possible because the tiles that make up the map in neatline are dynamically pulled by the javascript in response to user input (wat a bummer.) I will definitely check out these other suggestions, though, as the software I have been using isn't getting anything downloaded completely correctly (it seems to miss css @import rules, for instance).

   I would still love to someday find a way to create a static version of a simple neatline project simply because maintaining an installed version of neatline/Omeka for a one page display seems a bit excessive, and over the years will take dozens of extra hours in developer time (mostly when it's time to move servers yet again.)

   Posted 4 years ago Permalink

RSS feed for this topic

Reply

You must log in to post.