From time to time there are discussions on LinuxQuestions or IRC about making SlackBuilds run faster using “multiple jobs” that will use all the cpus in your system.

The good news is that it usually works. You can tell GNU make to use “multiple jobs” in parallel by setting an environment variable (see below). Most upstream software is built using GNU make, including all the builds that take a very long time, so this solution works well.

The bad news is that, for some upstream software, it sometimes fails.

How do I build with multiple jobs?

For SlackBuilds from SlackBuilds.org, this will work in most circumstances (the nproc command returns the number of cpus in your system).

export MAKEFLAGS="-j$(nproc)"

Some of the build tools, and other SlackBuilds repositories, may have other options.

  • For sbopkg you can set MAKEFLAGS in sbopkg.conf
  • For sbotools you can set ‘-j’ with sboconfig
  • For many Slackware SlackBuilds you can set the NUMJOBS environment variable
  • For many slacky.eu SlackBuilds you can set the JOBS environment variable

Note that there will be some stages of the build where not all of your cpus will be used, for example “configure” at the beginning, and linking and package creation at the end.

Don’t use too many jobs. It’s not really a problem to have many more jobs than cpus, but each additional job will need more and more memory. Eventually you will run out of memory and the build will fail.

GNU make also has a new alternative to MAKEFLAGS="-jN". For example, if you specify MAKEFLAGS="-l2.5", GNU make will only start more jobs if the system’s load average is less than 2.5.

Why does it fail sometimes?

Failures happen when output from one job is used as input by another job that doesn’t wait properly. If the first job hasn’t finished, the second job will fail. Running with only one job prevents the problem.

Sometimes a build with multiple jobs will fail on one system, even if it is ok on another system. It depends on the characteristics of the system, including the number of cpus, hyperthreading, cache sizes, storage speed, etc.

Success or failure sometimes changes, as bugs in upstream Makefiles and in GNU make get fixed and new bugs appear.

Some upstream software simply hasn’t been written to use multiple make jobs.

What errors will I see when a build fails with multiple jobs?

  • Errors about missing files or directories
  • Errors linking libraries that don’t exist
  • Builds that go very slow and then run out of memory

Writing SlackBuilds for multiple jobs

If you are writing a SlackBuild for SlackBuilds.org, please do not do anything.

  • If you don’t do anything, the user will be able to use the MAKEFLAGS environment variable, and the options in sbopkg, sbotools, etc.
  • But if your SlackBuild fails with multiple jobs, please use a hardcoded make -j1 command. You might also need make -j1 install.
  • Sometimes you will see make || make -j1 (that is, if make fails, try again with ‘-j1’). This might confuse inexperienced users, and might annoy people with slow systems, so please don’t use this code unless you specifically need it.
  • Please don’t invent some new environment variable instead of MAKEFLAGS.
  • Please don’t copy the code from Slackware’s SlackBuilds. The default is good for Patrick’s own professional build system and workflow (he uses ‘-j7’), but it’s not good for people with old or small systems.
  • For the same reason, please don’t hardcode something like ‘-j3’. It might be good for you, but it’s not good for everybody.
  • Please don’t calculate the “right” number of jobs from the number of CPUs. The user might want to do something else while the build is running in the background, or might be worried about overheating, or might want to use ‘-l’ instead of ‘-j’.

Which SlackBuilds fail with multiple jobs?

SlackBuilds from SlackBuilds.org that almost always fail with multiple jobs usually have ‘-j1’ hardcoded in them, so they will build even if you have set the MAKEFLAGS environment variable.

But because everybody’s system is different, and because bugs come and go, it is very difficult for the SBo maintainers and admins to set ‘-j1’ in all the SlackBuilds that need it.

Here’s a list of SlackBuilds that sometimes seem to fail with multiple jobs for me. This list is for building on Slackware 14.2.

academic/abntex
academic/arpack
academic/gretl
academic/staden
academic/tophat
academic/units
audio/Pd-extended
audio/dssi-vst
audio/flake
audio/non
desktop/ClipboardViewer
desktop/afterstep
desktop/avant-window-navigator
desktop/ee
desktop/ion
desktop/lumina
desktop/menumaker
development/bmake
development/camlp5
development/chibi-scheme
development/chicken
development/ecl
development/gauche
development/gengetopt
development/gforth
development/gnats
development/google-go-lang
development/gprolog
development/icon
development/latrace
development/menhir
development/ming
development/nvi
development/polyml
development/regina-rexx
development/rmlmmc
development/simulavr
development/smlnj
development/sourcenav
development/splint
games/adl
games/armagetronad
games/dopewars
games/ember
games/lmpc
games/megaglest
games/openmsx-catapult
games/pinball
games/tome4
gis/saga-gis
graphics/tuxpaint
libraries/agg
libraries/libb64
libraries/librfm5
libraries/stfl
libraries/xmlrpc-c
libraries/xview
multimedia/dvdrip
multimedia/lives
network/dsniff
network/gtk-gnutella
network/guacamole-client
network/milter-greylist
network/openl2tp
network/sslh
network/thrift
network/tightvnc
office/color-theme
office/minidjvu
office/wyrd
system/ZoneMinder
system/collectd
system/dieharder
system/dvdisaster
system/dynamips
system/efitools
system/freeswitch
system/heirloom-doctools
system/hercules
system/hwinfo
system/icinga-web
system/mdocml
system/munin-node
system/opam
system/passwordsafe
system/rhash
system/syslog-ng
system/uae
system/unison
system/xen