Dumping ED: howto

These are notes on how to dump and import ED. This howto assumes you are running a *NIX os.

Getting the dump

• If you don't have already, install Python 3.
• Install lxml and any other Python modules required. The dump tool should scream at you if you are missing something.
• You need the Mediawiki dump tools. Either grab them from my fork or the original ones. My fork saves the image list so you can resume grabbing image urls. By grab I mean git clone.
• In a terminal, go to the cloned directory.
• The first time, use the following command:

python3 dumpgenerator.py --xml --images --api https://encyclopediadramatica.online/api.php

• This should create a new dump folder, named like this: encyclopediadramaticaonline-XXXXXXXX-wikidump/.
• Whenever you resume, use the following command:

python3 dumpgenerator.py --xml --images --resume --path=encyclopediadramaticaonline-XXXXXXXX-wikidump/ --api https://encyclopediadramatica.online/api.php

Be sure to replace the X's with the aprropriate date.

• While dumping the xml, or the images, the wiki might throw an error, or you might temporarily lose connection. If you do not want to babysit the dumping script, use the following command:

until ./resume.sh; do
	echo "crashed with code $?. Respawning in 1800s..." >&2
	sleep 1800
	echo "finished waiting, respawning now"
done

resume.sh contains python3 dumpgenerator.py --xml --images --resume --path=encyclopediadramaticaonline-XXXXXXXX-wikidump/ --api https://encyclopediadramatica.online/api.php, obviously.
This process can take whole days to complete. Afterwards, the folder encyclopediadramaticaonline-XXXXXXXX-wikidump weighs approximately 56 gigabytes (as of October 2023), and contains everything you need to restore the wiki.

Importing the wiki

This is the more technical part. Don't hesitate to get help from OpenAI. In order to run a mediawiki, you will need to install and enable mariadb and apache2 services. You will need to install php8 and might need lua5.1 (you can try different php versions, however if you need lua, it must be 5.1).
Installing a mediawiki can differ a lot from system to system and is left as an exercise to the reader, the process is very well documented on the mediawiki wiki.

• How to install mediawiki on archlinux
• How to install mediawiki on ubuntu

Tip: every time you modify a config or install a missing dependency, make sure to restart the whole apache service.
Tip: on certain systems, the apache service is called httpd.
Steps to take once you have set up a blank wiki instance:
Configuring the LocalSettings.php file:

• Add the following after the $wgServer line: $logoPath = "logos";.
• Change the $wgLogos line to

$wgLogos = [
	'1x' => "$wgResourceBasePath/$logoPath/Encyclopedia_Dramatica_logo.svg",
	'icon' => "$wgResourceBasePath/$logoPath/Encyclopedia_Dramatica_logo.svg",
];

• Add the following right after:

$wgFooterIcons["poweredby"]["mediawiki"] = [
	"src" => "$wgResourceBasePath/$logoPath/powered_by_drama.png",
	"url" => "$wgServer",
	"alt" => "Powered by Drama",
];
$wgFavicon = "$wgResourceBasePath/$logoPath/ED_favicon_saber.png";

• Change the $wgUseImageMagick = true; line to $wgUseImageMagick = false;. Note, this might not be necessary. If images are not broken, you can leave this at true.
• Add the following to the end of the file:

wfLoadExtension( 'Scribunto' );
$wgScribuntoDefaultEngine = 'luastandalone';
$wgScribuntoEngineConf['luastandalone']['luaPath'] = '/usr/bin/lua5.1';
wfLoadExtension( 'SyntaxHighlight_GeSHi' );
wfLoadExtension( 'ParserFunctions' );
$wgPFEnableStringFunctions = true;
$wgScribuntoUseGeSHi = true;
wfLoadExtension( 'YouTube' );
wfLoadExtension( 'TemplateStyles' );
wfLoadExtension( 'MobileFrontend' );

• You may need to edit $wgScribuntoEngineConf accordingly to your setup.
• Download and extract the following extensions to their respective folders:
  • Scribunto
  • SyntaxHighlight
  • ParserFunctions
  • YouTube
  • TemplateStyles
  • MobileFrontend
• Note: you will need some of these extensions before running the import scripts, so be sure to follow in order.
• Copy the logos folder (the one with the images) to the root of your mediawiki installation.

Performing the import:

• Open a terminal and go to the mediawiki folder (the one which contains LocalSettings.php).
• Use the following command to import articles and edits (be sure to replace the example path with the actual xml path):

cd maintenance && php run.php importDump --conf ../LocalSettings.php /path/to/dump/encyclopediadramaticaonline-XXXXXX-wikidump/encyclopediadramaticaonline-XXXXXX-history.xml  --username-prefix="" --report 100

• Once this is done, you can import the images using the following command:

cd maintenance && php run.php importImages.php /path/to/dump/encyclopediadramaticaonline-XXXXXXXX-wikidump/images --comment-ext=desc

• Note: importing the xml might take a couple days. Importing images can take a day too. You can interrupt and resume the scripts with no problems.

General note: these instructions are not set in stone and will inevitably rot. Use your brain and adapt. Ask people and the bot.

~ jack