02 Aug Hidden Docs and Functionality in macOS
Overview
There are a multitude of useful command-line utilities that come with macOS. After using them day in and day out, a few of you may even know some of these tool inside and out, have all the documentation memorized, and even have written scripts around them to compensate for a few short-comings. What if there were a way to get a little more from these tools?
I am going to go over some of the techniques I have used to find some extra functionality in macOS. I’ve outlined a few of the tools and the general process I have used over the years to dig around macOS command-line utilities looking for anything useful. There are some pretty cool things you can uncover, but the first steps are knowing “what, where, and how.”
The What – Discovering File Types
file is a tool that can be used to read the magic number of files to find out what type of file it is:
bash$> cd /usr/sbin/ bash$> file snmpd vpnd vifs snmpd: POSIX shell script text executable, ASCII text vpnd: Mach-O 64-bit executable x86_64 vifs: Mach-O universal binary with 2 architectures: [i386: Mach-O executable i386] [x86_64: Mach-O 64-bit executable x86_64] vifs (for architecture i386): Mach-O executable i386 vifs (for architecture x86_64): Mach-O 64-bit executable x86_64
Using file, we can easily know what kind of file type we are looking at (regardless of its file extension), and knowing is half the battle. If there isn’t any magic number for a file, you’ll simply get the output data, which is not particularly useful.
The Where – Finding Binaries
It’s one thing to know what type of file you have, but you also have to know where they are located. If you want a large list of executable binaries from a system, this is one comprehensive method I would suggest:
bash$> find $(echo $PATH | tr ':' ' ') -type f -perm -111 2> /dev/null
To provide a breakdown, this one-liner essentially uses $(echo $PATH | tr ‘:’ ‘ ‘) to split up your $PATH environment variable into its separate paths. It uses -perm -111 to only print items with the executable bit set for owner, group, and everyone. The -type f flag is used to only print files (i.e. exclude directories), and finally, we redirect any errors the command may encounter along the way with 2> /dev/null.
I feel like I used to have a script that did this using the which command, but it seems to have been lost to time, and probably for the best. find is a very powerful tool and I felt it was worth demonstrating some of its versatility.
If you want to take it a step further, you could pipe the output of this command intoxargs file to print out the type of each file encountered:
bash$> find $(echo $PATH | tr ':' ' ') -type f -perm -111 2> /dev/null | xargs file
Which would effectively give you the complete path, as well as they type of each and every executable available for tab completion in Terminal.
However, if you’re only interested in a particular directory, you can replace $(echo $PATH | tr ‘:’ ‘ ‘) with any path and will get a list of every executable file in that directory (assuming you have the appropriate permissions to traverse the directory).
bash$> find /usr/sbin -type f -perm -111 2> /dev/null | xargs file ... /usr/sbin/KernelEventAgent: Mach-O 64-bit executable x86_64 /usr/sbin/kextcache: Mach-O 64-bit executable x86_64 /usr/sbin/kextfind: Mach-O universal binary with 2 architectures: [x86_64: Mach-O 64-bit executable x86_64] [i386: Mach-O executable i386] /usr/sbin/kextfind (for architecture x86_64): Mach-O 64-bit executable x86_64 /usr/sbin/kextfind (for architecture i386): Mach-O executable i386 /usr/sbin/kextlibs: Mach-O universal binary with 2 architectures: [x86_64: Mach-O 64-bit executable x86_64] [i386: Mach-O executable i386] /usr/sbin/kextlibs (for architecture x86_64): Mach-O 64-bit executable x86_64 /usr/sbin/kextlibs (for architecture i386): Mach-O executable i386 /usr/sbin/kextstat: Mach-O universal binary with 2 architectures: [x86_64: Mach-O 64-bit executable x86_64] [i386: Mach-O executable i386] /usr/sbin/kextstat (for architecture x86_64): Mach-O 64-bit executable x86_64 /usr/sbin/kextstat (for architecture i386): Mach-O executable i386 /usr/sbin/krbservicesetup: Mach-O 64-bit executable x86_64 /usr/sbin/ktutil: Mach-O 64-bit executable x86_64 /usr/sbin/languagesetup: Mach-O 64-bit executable x86_64 /usr/sbin/localemanager: Mach-O 64-bit executable x86_64 ...
Now that you’ve found them, it’s time to read them.
The How – Sifting through the Data
When it comes to binaries, cat is not an option, everything is just interpreted by Terminal including control sequences (alert, backspace, vertical tab, etc.) which effectively feels like a electronic war zone, but don’t be discouraged! There is always another way!
strings is an incredibly useful tool for getting the printable information from binaries while ignoring the rest. Although they are compiled, hard-coded strings are left intact, so you can get at a lot of error messages and (most importantly) any help documentation that is included. Granted, this is not a catch-all for all finding undocumented features, but it is a great place to start.
There CAN be a lot to sift through though, and not all of it is enlightening. The output from strings /usr/sbin/diskutil alone is 2625 lines (as of macOS 10.12).
Other tools for consideration are: hexdump,otool,nm, and od. However, it’s a little out of the scope of this post to outline the use of these tools here.
A Good Example – Apple Software Restore
The most useful item I have found using strings is with asr (Apple Software Restore). See man (8) asr.
Typical Usage:
bash$> asr help
Usage: asr <verb> <options>
<verb> is one of the following:
asr help | version
asr restore --source <source> --target <target> [<options>]
asr restore --source asr://<host>/ --file <file> [<options>]
asr server --source <source> --config <plist> [<options>]
asr imagescan --source <source> [--filechecksum] [--nostream]
asr info --source <source> [--plist] <size> is in bytes but may end with a scale factor (b, k, m, g)
common <options> are any of:
--source <source> path or url to disk image file, mountpoint, or
web accessible disk image
--puppetstrings print out messages in format good for machine parsing
--verbose display verbose output
--debug display debug output
restore <options> are any of:
--target <target> path to volume or mountpoint
--erase formats target volume
--format <HFS+|HFSX> target format when erasing (defaults to source)
--noprompt don't require confirmation on erase
--noverify don't checksum results
--buffers <num> number of buffers to use in block copy
--buffersize <size> size of buffers to use in block copy
--csumbuffers <num> number of buffers for the checksum if different
--csumbuffersize <size> size of buffers for the checksum if different
--timeout <seconds> max wait for stream in multicast client mode
--SHA256 force asr to verify with a SHA2-256 hash
server <options> are any of:
--interface <if> Use 'if' as the interface for the server's
outgoing stream
--config <configuration> server configuration file in plist format
imagescan <options> are any of:
--filechecksum calculate file checksum
--nostream don't reorder file for multicast streaming
Hidden Usage:
bash$> strings /usr/sbin/asr
...
common <options> are any of:
--source <source> path or url to disk image file, mountpoint, or
web accessible disk image
--puppetstrings print out messages in format good for machine parsing
--verbose display verbose output
--debug display debug output
restore <options> are any of:
--target <target> path to volume or mountpoint
FULL_USAGE
--hidden restore to the hidden customer software partition
--erase formats target volume
--format <HFS+|HFSX> target format when erasing (defaults to source)
--noprompt don't require confirmation on erase
--noverify don't checksum results
--buffers <num> number of buffers to use in block copy
--buffersize <size> size of buffers to use in block copy
--csumbuffers <num> number of buffers for the checksum if different
--csumbuffersize <size> size of buffers for the checksum if different
--timeout <seconds> max wait for stream in multicast client mode
--SHA256
force asr to verify with a SHA2-256 hash
server <options> are any of:
--interface <if> Use 'if' as the interface for the server's
outgoing stream
--config <configuration> server configuration file in plist format
imagescan <options> are any of:
--filechecksum calculate file checksum
--nostream don't reorder file for multicast streaming
--partnumber <string> add string to metadata during imagescan (key "part number")
Usage: %s <verb> <options>
<verb> is one of the following:
%s help | version
%s restore --source <source> --target <target> [<options>]
%s restore --source asr://<host>/ --file <file> [<options>]
%s server --source <source> --config <plist> [<options>]
%s imagescan --source <source> [--filechecksum] [--nostream]
%s info --source <source> [--plist]
%s partition --target <whole disk> [--testsize <size>]
[--retestsize <size>] [--recoverysize <size>]
%s freeze --target <target partition/volume> [--testsize <size>]
[--retestsize <size>] [--recoverysize <size>]
%s thaw --target <volume> [--recovery] [--modifyrecovery]
%s adjust --target <partition> [--settype <partType>]
...
This chunk of text is roughly 800 lines into the output of strings /usr/sbin/asr, but highlighted above, plain as day, we have extra functionality as well as usage.
Once found, hidden features still take a significant amount of testing and experimentation to utilize properly. Because of their nature, there typically isn’t a whole lot of documentation (go figure). What you see is what you get. Sometimes it’s pretty obvious, other times it isn’t at all. You’ll have to experiment and reverse engineer each executable to discover the purpose of the features for yourself.
According to Der Flounder’s ASR’s Hidden Documentation, there are other ways at getting to this specific chunk of documentation:
bash$> FULL_USAGE=1; asr help
tcsh#> setenv FULL_USAGE 1; asr help
However, you can’t rely on the incorporation of a universal FULL_USAGE environment variable for all compiled executables… Then again, it might be fun to find out if any other macOS binaries use this flag.
Other Stuff
Here are some other hidden features I have found using this method:
scutil:
--disable-until-needed
--allow-new-interfaces
networksetup:
-isHostNameValid
-isComputerNameValid
Conclusion
I wish I had a super awesome script to give that would just uncover all of the hidden flags of our most used utilities, but unfortunately it’s not that easy. I discovered this hidden usage of asr a long time ago using this method, and tried to find a few others for this post, but it was hard work. It took me about 4 hours of reading very abstract inline help strings as well as testing to find the few examples I provided here (I didn’t want to leave you empty handed).
It’s a process, and a pretty involved one at that, but hopefully it will lead to some cool discoveries. Happy hunting!
Links
Gary Kessler’s: File Signatures
Der Flounder’s: ASR’s Hidden Documentation
No Comments