scripts for console jockeys

I’ve found that in times of travel or of tight-deadline coding, my usage of DT diminishes – largely because I start using my laptop a lot. Despite having a mac at home and in the office, my laptop is a an ultraportable loaded with linux… the macbooks may be nice, but they can’t beat a 2-pound, 1-inch-thick machine for portability.

I use rsync and cvs over ssh to keep the laptop synchronized with the two macs. The only thing I’ve been missing is the ability to work with DT remotely: add files, export records [particularly emails with design notes], search for strings.

To make a long story short, I picked up an applescript book this week and cranked out some shell scripts that use osascript to communicate with DT.

dt_contents is used to print the contents of the database; it is a bit slow, and can be sped up by only printing subtrees. The -v flag shows record details such as the ID.

#!/bin/sh
# dt_contents.sh: Output the names of all [non-group] records in a DT database.
#                 Options allow specifing the top-level group for the listing
#                 and enable verbose output.

HELP_STR="Usage: $0 [-v] [-p path]"
START_PATH="root of current database"
DETAIL=""
DL="\"|\""	# delimiter

while getopts v\?p: opt
do
	case "$opt"
	in
		# NOTE: 'size' property fails sometimes so we use word count
		v) DETAIL="$DL & (id of c) as text & $DL & kind of c & $DL & \
			   date of c & $DL & (word count of c) as text & \
			   $DL & comment of c &";;
		p) START_PATH="(get record at \"$OPTARG\" in current \
		               database)" ;;
		\?) echo "$HELP_STR";  exit 1;;
	esac
done

# Applescript for contents
VAR=`osascript <<EOF
set nl to "\\\\\\\\n"	-- newline (escaped)

on do_group(g)
    set out to ""
    
    tell application "DEVONthink Pro"
        set g_children to children of g
    end tell

    repeat with c in g_children

        tell application "DEVONthink Pro"
            if type of c is group then
                set recurse to true
             else 
		set recurse to false
		set out to out & location of c & name of c & $DETAIL my nl
             end if
        end tell

        if recurse then
             set out to out & do_group(c)
	end if
	
    end repeat

    return out
end

tell application "DEVONthink Pro"
    copy $START_PATH to r
end tell

try
    -- verify that path exists: nested try is an attempt at 'if defined(r)'
    get r
    try
	set result to do_group(r)
    on error msg
        set result to "ERROR: " & msg & nl
    end try
on error
    set result to "ERROR: Invalid path" & nl
end try


EOF`

echo -ne $VAR

The dt_search script is used to find matching records based on the DT ‘search’ routine:

#!/bin/sh
# dt_search.sh : Search DT database for a string. Options allow the top-level
#                group for the search, and provide access to the standard
#                DT search parameters (comparison, operator, within).

HELP_STR="Usage: $0 -[E|F|I] -[A|P|W|Z] -[a|c|p|t|u] [-g path] search_string
	Comparison Options
	   -E       Exact (default)
	   -F       Fuzzy
	   -I       Case-insensitive
	Operator Options
	   -A       Any word
	   -P       Phrase
	   -W       Wildcards
	   -Z       All words (default)
	Within Options
	   -a       All (default)
	   -c       Comments
	   -p       Paths
	   -t       Titles
	   -u       URLs
	Other Options
	   -g path  Search within group at 'path'"

SEARCH_CMP="exact"
SEARCH_OP="all words"
SEARCH_IN="all"
SEARCH_GROUP=""

while getopts EFIZAPWacptu\?g: opt
do
	case "$opt"
	in
		# Comparison Flags
		E) ;;
		F) SEARCH_CMP="fuzzy" ;;
		I) SEARCH_CMP="no case" ;;
		# Operator Flags
		Z) ;;
		A) SEARCH_OP="any word" ;;
		P) SEARCH_OP="phrase" ;;
		W) SEARCH_OP="wildcards" ;;
		# Within Flags
		a) ;;
		c) SEARCH_IN="comments" ;;
		p) SEARCH_IN="paths" ;;
		t) SEARCH_IN="titles" ;;
		u) SEARCH_IN="URLs" ;;
		# Group to search in
		g) SEARCH_GROUP="in (get record at \"$OPTARG\")" ;;
		\?) echo "$HELP_STR";  exit 1;;
	esac
done

if [ $((OPTIND)) -gt $# ]
then
	echo "ERROR: search_string argument is mandatory"
	exit 2
fi

# Get Params
shift $((OPTIND - 1))
SEARCH_STR=$1

# Input validation
if [ -z "$SEARCH_STR" ]
then
	echo "ERROR: search_str cannot be empty"
	exit 3
fi

# Output format
DL="\"|\""
DETAILS="location of r & name of r & $DL & (id of r) as text & $DL & \
         kind of r & $DL & date of r & $DL & (word count of r) as text & \
         $DL & comment of r"

# Applescript for search
VAR=`osascript <<EOF
set nl to "\\\\\\\\n"	-- newline (escaped)

on find_records(s)
    set out to ""
     
    tell application "DEVONthink Pro"
	search s comparison $SEARCH_CMP $SEARCH_GROUP operator $SEARCH_OP within $SEARCH_IN 
	
	repeat with r in result
	    set out to out & $DETAILS & my nl
	end repeat
    end tell

end

set result to find_records("$SEARCH_STR")

EOF`

echo -ne $VAR

The dt_export script is fairly straightforward, and should give an idea how to do imports as well:

#!/bin/sh
# dt_export.sh : Export a record from a DT database based on its ID.
#                Path to which the record will be exported is required.

HELP_STR="Usage: $0 record_id output_path"

while getopts \? opt
do
	case "$opt"
	in
		\?) echo "$HELP_STR";  exit 1;;
	esac
done

if [ $((OPTIND + 1)) -gt $# ]
then
	echo "ERROR: record_id and output_path arguments are mandatory"
	exit 2
fi

# Get Params
shift $((OPTIND - 1))
REC_ID=$1
REC_PATH=$2

# Input validation
echo $REC_ID | grep '[^0-9]'
if [ $? -eq 0 ]
then
	echo "ERROR: record_id must be an integer"
	exit 3
fi

if [ -z "$REC_PATH" ]
then
	echo "ERROR: output_path must be non-NULL"
	exit 4
fi

# Applescript for export
VAR=`osascript <<EOF
set nl to "\\\\\\\\n"	-- newline (escaped)

on export_record(r_id, out_path)
     
    tell application "DEVONthink Pro"
        set r to get record with r_id
		-- check that record exists
        try
	    	get r
		on error msg
	    	return "ERROR: Unable to find " & e_id as text & ": " & msg & my nl
		end try
		
		-- export record
		try
	    	export record r to out_path
		on error msg
	    	return "ERROR: Unable to write " & out_path & ": " & msg & my nl
		end try
    end tell

end

set result to export_record($REC_ID, "$REC_PATH")

EOF`

echo -ne $VAR

The dt_take_note script is somewhat more interesting; it sends STDIN to the clipboard, then pastes the text into DT as a note. Very useful for stuff like “grep $PAT log | dt_take_note.sh /Logs”.

#!/bin/sh
# dt_take_note : Take plain text on STDIN and import it into DT at
#                the specified location.

HELP_STR="Usage: $0 [path]"

while getopts \? opt
do
	case "$opt"
	in
		\?) echo "$HELP_STR";  exit 1;;
	esac
done

# Get Params
shift $((OPTIND - 1))
DEST_PATH="(get record at \"$1\")"

if [ -z "$DEST_PATH" ]
then
	DEST_PATH="(get root of current database)"
fi

# Read in data from STDIN
DATA=""
while read LINE
do
	DATA="${DATA}${LINE}\n"
done
echo -e $DATA | pbcopy


# Applescript for record info
VAR=`osascript <<EOF

on take_note()
    set nl to "\\\\\\\\n"	-- newline (escaped)

    tell application "DEVONthink Pro"
        set g to $DEST_PATH
		-- check that dest group exists
        try
	    	get g
		on error msg
	    	return "ERROR: Unable to find " & location of g & name of g & ": " & msg & nl
		end try
		
		-- take note
		set r to paste clipboard to g 
		set out to "ID: " & (id of r) as text & "|" & location of r & name of r & nl
    end tell

	return out
end

set result to take_note()

EOF`

echo -ne $VAR

I also created the following scripts whose implementation should be obvious from the above [no need to make this post even more ridiculously long]:

# dt_import.sh : Import a file into a DT database. Options allow the
#                type of file and the destination group to be specified.

# dt_groups.sh : List groups in the current Devonthink Database.
#                Options allow recursive listing and specifying the top-level 
#                group for the listing.

# dt_classify : Output classification proposals for a DT record base on 
#              its database ID.

# dt_record_info : Output details of a DT record base on its database ID.

These cover the basics of DT usage, and mean I’ll be able to use DT remotely [in the Terminal, over ssh] when I travel again this week.

I’m curious if any other command-line types have ideas for similar scripts, or more ambitious projects. It would be nice, for example, to be able to synchronize a DB on two machines using cvs. A DB module for python or ruby would be quite interesting, though possibly overkill.

–Eric

Very, VERY nice!!!

Thanks for sharing. I’m sure these will prove very useful for me and many others here as well.

Thanks!

Here’s the code for dt_record; I shouldn’t have left it out, as it turns out to be very useful.

For example:


for i in `dt_search.sh -I $SEARCH_TERM | cut -d\| -f 2`; do    dt_record_info.sh $i; done

The code itself is:

#!/bin/sh
# dt_record_info : Output details of a DT record base on its database ID.

HELP_STR="Usage: $0 record_id"

while getopts \? opt
do
	case "$opt"
	in
		\?) echo "$HELP_STR";  exit 1;;
	esac
done

if [ $((OPTIND)) -gt $# ]
then
	echo "ERROR: record_id argument is mandatory"
	exit 2
fi

# Get Params
shift $((OPTIND - 1))
REC_ID=$1

# Input validation
echo $REC_ID | grep '[^0-9]'
if [ $? -eq 0 ]
then
	echo "ERROR: record_id must be an integer"
	exit 3
fi


# Applescript for record info
VAR=`osascript <<EOF

on record_info(r_id)
    set nl to "\\\\\\\\n"	-- newline (escaped)

    tell application "DEVONthink Pro"
        set r to get record with r_id

		-- check that record exists
        try
	    	get r
		on error msg
	    	return "ERROR: Unable to find " & e_id as text & ": " & msg & nl
		end try
		
		-- attempt to get size [broken!]
		try
			set sz to size of r
		on error
			set sz to 0
		end try 
		
		-- output record info
		set out to "ID: " & (id of r) as text & nl
		set out to out & "Type: " & kind of r & " Size: " & sz as text & " Word Count: " & (word count of r) as text & nl
		set out to out & "Duplicates: " & (number of duplicates of r) as text & " Replicants: " & (number of replicants of r) as text & nl
		set out to out & "Create Date: " & creation date of r & " Modify Date: " & modification date of r & nl
		set out to out & "Location: " & location of r & name of r & nl
		set out to out & "Path: " & path of r & filename of r & nl
		set out to out & "URL: " & URL of r & nl
		set out to out & "Wiki Aliases: " & aliases of r & nl
		set out to out & "Label: " & (label of r) as text & " Comment: " & comment of r & nl
		set out to out & "Script: " & attached script of r & nl
		set out to out & "Source: " & source of r & nl
		set out to out & "Text: " & plain text of r & nl
    end tell

	return out
end

set result to record_info($REC_ID)

EOF`

echo -ne $VAR

I’m experiencing a ‘morning after’ reacquaintance with the scripts, and am finding them very useful… in particular dt_search, dt_take_note, and dt_record_info. I tend to prefer the command line for doing things, and am finding records much more quickly this way [less mousing around].

If anyone more familiar with The DT Way has ideas for other shell scripts, let me know – I’ll be happy to give writing them a shot.

I’m thinking that a script to set record metadata [e.g. label,comment] will be extremely useful… the output of dt_search could then be munged to produce a list of record IDs which will be assigned specific metadata. I guess I’ve found tonight’s project :wink:

–Eric

Thanks^3 for these cool scripts!

The output from dt_contents is much more usable to me than Export > Listing… output because it includes full item paths and uses Unix newlines. And dt_record_info output is helpfully thorough.

I can imagine combining those two scripts into one for comparing content info of two databases, possibly helping isolate differences like I wrote about earlier today. Basically, it could be verbose output for dt_contents that includes relevant dt_record_info output (e.g. number of duplicates and replicants) in a format suitable for diff-like comparison when it’s run it on each database. Replicants are a a bit tricky because they don’t necessarily have the same pathname in comparable databases, though only their dirname differs and the basename is identical.

I think that kind of db comparison functionality would have some general benefits beyond my own specific needs, such as running it on the same database at regular intervals and comparing results as an additional integrity checker. And there might be other value in having a convenient summary of changes over time.

That’s it for the moment – other plans for the evening are calling. Thanks again!

This one is a straight applescript.

I have many binary files (in-house format) which I would like to add to DT, along with numerous perl and python scripts to convert the files to plain text (usually csv). The need for custom input handlers has been expressed before:

http://www.devon-technologies.com/phpBB2/viewtopic.php?t=3073

This script operates only on DT Link records (files which could not be indexed). It prompts the user for a command to use for the conversion, and creates a new Plain Text record in the same location (with the same name) as the Link record containing the output of the command.

It also associates commands with file types and extensions – see the header of the script for details.


-- Convert Using Command:
-- Create a Plain Text record of a Linked (i.e., unrecognized by DT)
-- file using a command specified by the user.
-- The command can take two forms:
--      cat path_to_file | command      # send contents of file to STDIN
--      command path_to_file            # pass file path as a parameter
-- The first form is the default behavior. To use the second form,
-- specify the path in the command with $FILENAME. The path of the
-- file will be substituted when the command is run, e.g.
--     "command -f $FILENAME" becomes "command -f path_to_file"
--
-- File Type Association: By default, File type associations are
-- enabled. This creates the top-level group __Config__. The
-- advantages of associations are that a command can be linked to
-- a file type (or extension) for repeated use. To disable file type
-- associations, set enable_file_type_assoc to 'false. To change the location
-- of the settings, modify the config_dir variable. Note that config_dir
-- and all file type association records are ecxcuded from classification.

-- Enable File Type Assoc :
-- global variable to enable association of commands with file types:
-- set to True to have associations stored in DT. 
global enable_file_type_assoc
set enable_file_type_assoc to true

-- Config Dir:
-- location of config directory in DT 
-- (used when enable_file_type_assoc is True)
global config_dir
set config_dir to "/__Config__"

-- location of file type associations in DT 
-- (used when enable_file_type_assoc is True)
global file_type_dir
set file_type_dir to config_dir & "/Convert Using Command/File Types"

-- most recent command string
-- (used when > 1 record is selected)
global cmd_str
set cmd_str to ""

-- most recent file type string
-- (used when > 1 record is selected)
global file_type
set file_type to ""


-- *** File Type Associations ***
on create_file_type_group()
	-- add group for file type commands
	tell application "DEVONthink Pro"
		if not (exists record at config_dir) then
			set grp to create location config_dir
			set exclude from classification of grp to true
		end if
		if not (exists record at file_type_dir) then
			set grp to create location file_type_dir
			set exclude from classification of grp to true
		end if
	end tell
end create_file_type_group

on set_file_type_cmd(file_type, cmd_str)
	-- set command for file type
	-- NOTE: this only overwrites previous association if record
	-- is not locked; file assoc records are locked on creation.
	create_file_type_group()
	set rec_path to file_type_dir & "/" & file_type
	
	tell application "DEVONthink Pro"
		if exists record at rec_path then
			-- overwrite existing association
			set rec to get record at rec_path
			if not locking of rec then set plain text of rec to cmd_str
		else
			-- create new association
			set grp to get record at file_type_dir
			set rec_date to current date
			set rec to create record with {name:file_type, type:txt, plain text:cmd_str, date:rec_date} in grp
			set exclude from classification of rec to true
			set locking of rec to true
		end if
	end tell
	
end set_file_type_cmd


on get_file_type_cmd(file_type)
	-- Get command for file type
	-- This is simply the plain text of the record with the same
	-- name as the file extension or type.
	-- If no association exists, "" is returned.
	
	create_file_type_group()
	set ftype_cmd to ""
	set rec_path to file_type_dir & "/" & file_type
	
	tell application "DEVONthink Pro"
		if exists record at rec_path then
			set rec to get record at rec_path
			set ftype_cmd to plain text of rec
			-- enforce single line of text: this has an embedded '\n'
			set idx to offset of "
" in ftype_cmd
			if idx > 0 then
				set ftype_cmd to texts from character 1 to (idx - 1) of ftype_cmd
			end if
		end if
	end tell
	
	return ftype_cmd
end get_file_type_cmd

-- *** Data Conversion *** --
on create_new_rec(rec, cmd_results, path_str, cmd_str)
	-- Create a new Plain Text record for the data in cmd_resultys
	-- rec is the current record, on which the new record is based
	-- cmd_results is the data to insert into the new record
	-- path_str is the path of the file from which the data was generated
	-- cmd_str is the command used to generate the file (for comment)
	tell application "DEVONthink Pro"
		set comment_str to "Created by 'Convert Using Command' Command:'" & cmd_str & "' Original record: " & id of rec as text
		set loc to location of rec
		set grp to get record at (texts from character 1 to ((length of loc) - 1) of loc)
		copy URL of rec to url_str
		copy name of rec to name_str
		set rec_date to current date
		create record with {name:name_str, type:txt, comment:comment_str, path:path_str, plain text:cmd_results, date:rec_date, URL:url_str} in grp
	end tell
end create_new_rec

-- *** Generate command for do shell script ***
on generate_cat_cmd(path_str, cmd_str)
	-- Generate a command of the form 'cat $FILENAME | command'
	return "cat " & path_str & " | " & cmd_str
end generate_cat_cmd

on generate_cmd(path_str, cmd_str, start_idx)
	-- Generate a command of the form 'command -f $FILENAME'
	-- This expands $filename to path of file; useful when 
	-- the command does not read from STDIN.
	set end_idx to start_idx + 9
	set cmd_start to text from character 1 to (start_idx - 1) of cmd_str
	if length of cmd_str > end_idx then
		set cmd_end to text from character end_idx to (length of cmd_str) of cmd_str
	else
		set cmd_end to ""
	end if
	return cmd_start & path_str & cmd_end
end generate_cmd

on perform_command(path_str, cmd_str)
	-- Generate the shell command for the conversion and execute it.
	-- Output of the command is returned as an ASCII string.
	set idx to offset of "$FILENAME" in cmd_str
	if idx = 0 then
		set shell_cmd to generate_cat_cmd(path_str, cmd_str)
	else
		set shell_cmd to generate_cmd(path_str, cmd_str, idx)
	end if
	
	try
		set cmd_results to do shell script shell_cmd
	on error number n
		tell application "DEVONthink Pro"
			log message "Convert Using Command" info "Command '" & shell_cmd & "' failed with exit code " & n as text
		end tell
		set cmd_results to ""
	end try
	
	return cmd_results
end perform_command

-- *** Get Command from user or from file type ***
on get_command(file_type)
	--	Get command to execute from user.
	--	If enable_file_type_assoc is set, the command associated with 
	--  the file type or extension is used as the default.
	
	if enable_file_type_assoc then
		set cmd_str to get_file_type_cmd(file_type)
	else
		set cmd_str to ""
	end if
	
	-- The dialog label has an embedded '\n'
	set dlg_str to "Enter the command used to convert the file to Plain Text.
Use $FILENAME to refer to path of file; otherwise file is sent to STDIN of command."
	display dialog dlg_str default answer cmd_str
	copy text returned of result to user_sel
	
	return user_sel
end get_command

-- *** Handle DT Link Record  ***
on get_file_type(path_str)
	-- Return registered file type of file at path_str, or the
	-- file extension of file is not of registered type. Default if the
	-- file has no extension is 'Text document'
	-- Returns "" only if the file does not exist
	tell application "Finder"
		set f to path_str as POSIX file
		if not (exists f) then return ""
		try
			set f_type to kind of f
		on error
			set f_type to name extension of f
			if f_type is "" then set f_type to "Text document"
		end try
	end tell
	
	return f_type
end get_file_type

on handle_record(rec)
	-- Get command for conversion and perform conversion
	tell application "DEVONthink Pro"
		if type of rec is not link then
			log message "Convert Using Command" info "Cannot process record '" & location of rec & name of rec & "': not a link"
			return
		end if
		
		copy path of rec to path_str
	end tell
	
	set ftype to get_file_type(path_str)
	if ftype is "" then
		tell application "DEVONthink Pro"
			log message "Convert Using Command" info "Cannot convert file '" & path_str & "': file does not exist"
		end tell
		return
	end if
	
	-- if file type is the same as previous file type
	-- (e.g. when processing a list of files), reuse cmd_str
	if file_type is not equal to ftype then
		set file_type to ftype
		set cmd_str to get_command(file_type)
	end if
	
	set cmd_results to perform_command(path_str, cmd_str)
	if cmd_results is "" then return -- already logged
	
	create_new_rec(rec, cmd_results, path_str, cmd_str)
	
	--	If enable_file_type_assoc is set, associate this command with file type 
	if enable_file_type_assoc then
		set_file_type_cmd(file_type, cmd_str)
	end if
	
end handle_record

-- *** main ***
tell application "DEVONthink Pro"
	set rec_list to the selection
end tell

if rec_list is {} then error "Please select at least one record"
repeat with rec in rec_list
	handle_record(rec)
end repeat

Note: I coded this up in Script Editor (I’m still not beyond debugging applescripts into existence, sadly) and it replaced all of my “\n” and “\t” characters with literal newlines and tabs. Keep an eye out for that – I’ve tried to mark in the code where they appear.

Coming up is an even longer one; I implemented the TextMate ‘Filter through Command’ feature that I earlier lamented the loss of…

–Eric

Another (long) pure applescript one, this time emulating TextMate’s “Filter through command” feature.

This script requires that some text or one or more records be selected. It then prompts the user for a command, and feeds the text or the contents of the record(s) to the command via STDIN (URLs are downloaded first, while Link and Media files are cat’ed based on the record Path property).

The user is also prompted for what to do with the output. The following options are provided:

  • Discard : Send results to /dev/null and write exit status to log
  • New record : Create a new plain text record with same name and location as current record
  • Append record : Append to plan text of current record
  • Replace record : Replace plain text of current record
  • Import as File : Import output into DT via a temporary file
  • Append comment : Append to comments of current record
  • Send to clipboard : send to os x clipboard
  • Send to log : write a message to DT log

Failures to produce output usually log a reason to the DT log window (e.g. when trying to replace the plain text of a Link record, which always fails).

Also, Import is broken – DT only creates Link records to the temp files. I use file(1) to determine the contents of the temp file, but DT appears to work on extension, so I’ll have to add a routine to change the extension of the temporary file before import.

Oh yeah, there is also a command history. This and the other script use /Config as a repository for script-related config/data files, though that can be disabled or changed.

Here is the code (see above post regarding the embedded newlines):


-- Filter Through Command :
-- Send selected text, record, or records to the specified command using 'do shell script'.
-- Different record types will be handled appropriately:
--    Text, RTF: Plain text of record is echoed to STDIN of command
--     HTML, XML: Source of record is echoed to STDIN of command
--     Link (file), Media files: file (path) of record is cated to STDIN of command
--     Link (URL): Markup of record is downloaded and echoed to STDIN of command
--     Sheet, Form: Cells are Tab-delimited and echoed to STDIN of command
--     Group: every record in group is handled as above
-- The user must select how the output is handled:
--     Discard : results are discarded, but exit status is written to DT log
--     Create : A new Text record is created from results with same name as input record
--     Append : Results are appended to the current record
--     Replace : Results overwrite the plain text of the current record
--   Import : Import as file (BROKEN)
--     Comment : Results are appended to the comment of the current record
--     Clipboard : Results are pasted to the clipboard
--     Log : Results are written to the DevonThink log
--
-- Command History: By default the command history is enabled. This 
-- created the top-level group __Config__. The advantage of a command
-- history is that previously-entered commands can be used without
-- retyping them. To disable the command history, set enable_cmd_history
-- to 'false'. To change the location of the settings, modify the config_dir variable.
-- Note that config_dir and all command history records are excluded from classification. 

-- TODO: Fix Output option 'import as file' e.g. for pdf etc output

-- Enable Command History:
-- global variable to enable history: set to True to have history stored in DT
global enable_cmd_history
set enable_cmd_history to true -- false

-- Config Dir:
-- location of config directory in DT 
-- (used when enable_cmd_history is True)
global config_dir
set config_dir to "/__Config__"

-- location of command history in DT 
-- (used when enable_cmd_history is True)
global cmd_history_dir
set cmd_history_dir to config_dir & "/Filter Through Command"

-- name of command history record 
-- (used when enable_cmd_history is True)
global cmd_history_file
set cmd_history_file to "Command History"

-- path to record containing command history in DT 
-- (used when enable_cmd_history is True)
global cmd_history_rec
set cmd_history_rec to cmd_history_dir & "/" & cmd_history_file

-- global variable for clipboard output
global clipboard_str
set clipboard_str to ""


-- *** Command History ***

-- add history group
on create_history_group()
	tell application "DEVONthink Pro"
		if not (exists record at config_dir) then
			set grp to create location config_dir
			set exclude from classification of grp to true
		end if
		if not (exists record at cmd_history_dir) then
			set grp to create location cmd_history_dir
			set exclude from classification of grp to true
		end if
		if not (exists record at cmd_history_rec) then
			-- create new history file
			set grp to get record at cmd_history_dir
			set rec_date to current date
			set rec to create record with {name:cmd_history_file, type:txt, plain text:"", date:rec_date} in grp
			set exclude from classification of rec to true
		end if
	end tell
end create_history_group

-- add to history
on add_to_history(cmd_str)
	-- Add command to history if it is not already there
	-- NOTE: this does not reorder commands when they are executed!
	-- TODO: use a group instead of file so we can sort by date
	create_history_group()
	tell application "DEVONthink Pro"
		set rec to get record at cmd_history_rec
		copy plain text of rec to hist_str
		set idx to offset of cmd_str in hist_str
		if idx is equal to 0 then
			-- if command is not in history, insert it at top of file
			if length of hist_str > 0 then
				-- delimit command with embedded newline '\n'
				set hist_str to "
" & hist_str
			end if
			set plain text of rec to cmd_str & hist_str
		end if
	end tell
end add_to_history

on get_command_list(hist_str)
	-- Returns a list of commands (one per line in history)
	if length of hist_str is equal to 0 then return {}
	
	-- split string based on embedded newline '\n'
	set AppleScript's text item delimiters to "
"
	set cmd_list to text items of hist_str
	set AppleScript's text item delimiters to ""
	
	return cmd_list
end get_command_list

-- show history listbox
on get_cmd_in_history()
	-- Returns the command history selected by the user.
	-- Returns an empty string if there are no items in history 
	-- or if user aborts.
	
	create_history_group()
	tell application "DEVONthink Pro"
		set rec to get record at cmd_history_rec
		copy plain text of rec to hist_str
	end tell
	
	set cmd_list to get_command_list(hist_str)
	if length of cmd_list is equal to 0 then
		display dialog "Command history is empty" buttons {"OK"}
		return ""
	end if
	
	set cmd_str to choose from list cmd_list OK button name "Select" cancel button name "Back"
	
	if cmd_str is false then set cmd_str to ""
	
	return cmd_str
end get_cmd_in_history

-- get most recent command
on last_cmd_in_history()
	-- Returns the top command in the history record or "".
	create_history_group()
	tell application "DEVONthink Pro"
		set rec to get record at cmd_history_rec
		copy plain text of rec to hist_str
	end tell
	
	if length of hist_str is equal to 0 then return ""
	
	-- search for first embedded newline "\n"
	set idx to offset of "
" in hist_str
	-- no newline? only one entry in history
	if idx is equal to 0 then
		set idx to length of hist_str
	else
		set idx to idx - 1
	end if
	
	set cmd_str to (text from character 1 to character idx of hist_str)
	
	return cmd_str
end last_cmd_in_history

-- *** Output Handlers ***
on handle_new_record(rec, output_data, cmd_str)
	-- Create a new text record with the same name and path as rec containing output_data
	-- Comment contains the name of this script, the coomand, and the 
	-- id of the original record
	tell application "DEVONthink Pro"
		set comment_str to "Created by: 'Filter Through Command' Command:'" & cmd_str & "' Original record: " & id of rec as text
		set loc to location of rec
		set grp to get record at (texts from character 1 to ((length of loc) - 1) of loc)
		copy path of rec to path_str
		copy URL of rec to url_str
		copy name of rec to name_str
		set rec_date to current date
		create record with {name:name_str, type:txt, comment:comment_str, path:path_str, plain text:output_data, date:rec_date, URL:url_str} in grp
	end tell
end handle_new_record

on handle_append_record(rec, output_data, cmd_str)
	-- 	Append output_data to end of plain text of rec data
	tell application "DEVONthink Pro"
		copy plain text of rec to text_str
		if length of text_str > 0 then set text_str to text_str & return
		try
			set plain text of rec to text_str & output_data
		on error msg
			log message "Filter through command '" & cmd_str & "'" info "Append plain text of record '" & (path of rec) & (name of rec) & "' failed: '" & (msg as text) & "'"
		end try
	end tell
end handle_append_record

on handle_replace_record(rec, output_data, cmd_str)
	-- 	Set plain text of record to output data. Logs an error on failure.
	tell application "DEVONthink Pro"
		try
			set plain text of rec to output_data
		on error msg
			log message "Filter through command '" & cmd_str & "'" info "Replace plain text of record '" & (path of rec) & (name of rec) & "' failed: '" & (msg as text) & "'"
		end try
	end tell
end handle_replace_record

on get_file_format(path_str)
	set format_strings to {"ASCII", "HTML", "Rich", "PostScript", "PDF"} --, "JPEG"}
	set file_magic to do shell script "file -b " & path_str & " | cut -f 1 -d ' '"
	tell application "DEVONthink Pro"
		set file_types to {simple, markup, rich, PDF, PDF} --, image}
		set f_type to all
		repeat with i from 1 to count of format_strings
			if file_magic = item i of format_strings then
				set f_type to item i of file_types
				exit repeat
			end if
		end repeat
	end tell
	return f_type
end get_file_format

on unlink_tmpfile(path_str)
	tell application "Finder"
		set f to path_str as POSIX file
		if exists f then delete f
	end tell
end unlink_tmpfile

on handle_import_file(rec, output_data, cmd_str)
	-- Import data as file (PDF, JPG, etc)
	-- remember, path to tmp file is passed in as output_data
	-- BROKEN: DT does not use magic to determine file type based on content... 
	--         probably have to fudge the file extension
	set tmpfile_path to output_data
	set tmpfile_fmt to get_file_format(tmpfile_path)
	
	tell application "DEVONthink Pro"
		set loc to location of rec
		set grp to get record at (texts from character 1 to ((length of loc) - 1) of loc)
		
		try
			set rec to import tmpfile_path to grp type tmpfile_fmt
			-- erase path of record since we are deleting tmp file
			set path of rec to ""
		on error msg
			log message "Filter through command '" & cmd_str & "'" info "Import as file '" & tmpfile_path & "' failed: '" & (msg as text) & "'"
		end try
	end tell
	
	-- remove temp file
	unlink_tmpfile(tmpfile_path)
end handle_import_file

on handle_append_comment(rec, output_data, cmd_str)
	-- 	Append output_data to comment of rec
	tell application "DEVONthink Pro"
		copy comment of rec to comment_str
		if length of comment_str > 0 then set comment_str to comment_str & " "
		set comment of rec to comment_str & output_data
	end tell
end handle_append_comment

on handle_clipboard(rec, output_data, cmd_str)
	-- 	Append output data to clipboard string
	set clipboard_str to clipboard_str & output_data
end handle_clipboard

on handle_log(rec, output_data, cmd_str)
	-- 	Append output_data to Devon Think log
	tell application "DEVONthink Pro"
		log message "Filter Through Command '" & cmd_str & "'" info output_data
	end tell
end handle_log

on handle_output(output_str, output_data, rec, cmd_str)
	-- Use output_str to determine output type, and invoke handler
	-- output_data is the actual text to output
	-- rec is the current record, used for appending/replacing/etc
	-- cmd_str is the command, currently only used by 'new record' and 'log'
	
	set output_strings to {"new", "append", "replace", "import", "comment", "clipboard", "log"}
	set output_handlers to {handle_new_record, handle_append_record, handle_replace_record, handle_import_file, handle_append_comment, handle_clipboard, handle_log}
	
	-- get output handler for output_str
	global out_handler
	set out_handler to handle_log
	repeat with i from 1 to count of output_strings
		if item i of output_strings is output_str then
			set out_handler to item i of output_handlers
			exit repeat
		end if
	end repeat
	
	out_handler(rec, output_data, cmd_str)
end handle_output

-- *** Filter Through Command ***
on perform_command(cmd_str, output_str, rec, orig_cmd)
	--	Execute command in cmd_str and invoke output handler
	-- cmd_str is the entire command to execute via do shell script
	-- output_str selects the output handler
	-- rec is the current record, for use by output handler
	-- orig_cmd is the original command (sans cat or echo) for logging
	
	if output_str is "discard" then
		-- use /dev/null for safe, fast discard
		set cmd_str to cmd_str & " > /dev/null"
	else if output_str is "import" then
		-- let the shell handle file creation instead of applescript
		-- NOTE this tmp file is removed in handle_import_file
		set tmp_path to "/private/tmp/DT_filter_thru_cmd_" & (random number (10000) as text)
		set cmd_str to cmd_str & " > " & tmp_path
	end if
	-- error cmd_str
	
	set status_code to 0
	try
		set cmd_results to (do shell script cmd_str)
	on error number n
		set status_code to n
	end try
	
	if output_str is "discard" or status_code is not equal to 0 then
		-- discard logs the exit code of the command, as does failure of shell script
		-- NOTE: output handler is not called on failure!
		set output_str to "log"
		set cmd_results to ""
		if status_code is not equal to 0 then
			set cmd_results to "Failed. "
			if output_str is "import" then
				-- remove temporary file created for import
				unlink_tmpfile(tmp_path)
			end if
		end if
		set cmd_results to cmd_results & "Exit code: " & (status_code as text)
	else if output_str is "import" then
		-- there are no results when using import; instead pass tmpfile path
		set cmd_results to tmp_path
	end if
	
	handle_output(output_str, cmd_results, rec, orig_cmd)
end perform_command

on perform_echo_command(input_str, rec, command_str, output_str)
	-- 	Echo input_str and pipe to to command_str
	-- NOTE: we use a shell HERE document with embedded newlines (\n)
	-- in the command to get around shell/applescript issues.
	set cmd_str to "cat <<__EOF__ | " & command_str & "
" & input_str & "
__EOF__
"
	perform_command(cmd_str, output_str, rec, command_str)
end perform_echo_command

on perform_cat_command(path_str, rec, command_str, output_str)
	-- 	Cat file and pipe to command_str
	set cmd_str to "cat " & path_str & " | " & command_str
	perform_command(cmd_str, output_str, rec, command_str)
end perform_cat_command

-- *** Input Handlers ***
on handle_text_rec(rec, command_str, output_str)
	-- 	ASCII or RTF: Get plain text of record and handle as text
	tell application "DEVONthink Pro"
		copy plain text of rec to text_str
	end tell
	perform_echo_command(text_str, rec, command_str, output_str)
end handle_text_rec

on handle_source_rec(rec, command_str, output_str)
	-- 	HTML or XML: get source of record and handle as text
	tell application "DEVONthink Pro"
		copy source of rec to source_str
	end tell
	perform_echo_command(source_str, rec, command_str, output_str)
end handle_source_rec

on handle_url_rec(rec, command_str, output_str)
	-- 	Link: Download url and handle as text
	tell application "DEVONthink Pro"
		copy URL of rec to url_str
		set html_src to download markup from url_str
	end tell
	perform_echo_command(html_src, rec, command_str, output_str)
end handle_url_rec

on handle_link_rec(rec, command_str, output_str)
	-- 	Link: Handle as path or as a url rec
	tell application "DEVONthink Pro"
		set path_str to path of rec
		copy path of rec to path_str
	end tell
	
	if path_str is "" then
		-- This is a link ot a URL
		handle_url_rec(rec, command_str, output_str)
	else
		-- This is an indexed file
		perform_cat_command(path_str, rec, command_str, output_str)
	end if
end handle_link_rec

on handle_media_rec(rec, command_str, output_str)
	-- 	Picture, Sound or Video: Handle as path
	tell application "DEVONthink Pro"
		copy path of rec to path_str
	end tell
	perform_cat_command(path_str, rec, command_str, output_str)
end handle_media_rec

on get_form_data(rec, delim)
	-- Return calls in record as a delimited string
	set data_str to ""
	tell application "DEVONthink Pro"
		copy cells of rec to cell_list
	end tell
	repeat with c in cell_list
		if data_str is not "" then set data_str to data_str & delim
		set data_str to data_str & (c as text)
	end repeat
	return data_str
end get_form_data

on handle_form_rec(rec, command_str, output_str)
	-- 	Form: Convert to tab-delim and handle as text
	set data_str to get_form_data(rec, "	") -- delim is TAB char '\t'
	perform_echo_command(data_str, rec, command_str, output_str)
end handle_form_rec

on handle_sheet_rec(rec, command_str, output_str)
	-- 	Sheet: Send to form handler
	tell application "DEVONthink Pro"
		set form_list to children of rec
	end tell
	set data_str to ""
	
	repeat with f in form_list
		tell application "DEVONthink Pro"
			set f_type to kind of f
		end tell
		if f_type = "record" then
			-- append newline to previous line -- embedded newline '\n'
			if data_str is not "" then set data_str to data_str & "
"
			set data_str to data_str & get_form_data(f, "	") -- delim is tab '\t'
		end if
		
	end repeat
	perform_echo_command(data_str, rec, command_str, output_str)
end handle_sheet_rec

on handle_group_rec(rec, command_str, output_str)
	-- 	Group of records: Recursively handle all records in group
	tell application "DEVONthink Pro"
		set rec_list to children of rec
	end tell
	repeat with rec in rec_list
		handle_record(rec, command_str, output_str)
	end repeat
end handle_group_rec

on handle_record(rec, command_str, output_str)
	-- Perform command on a single DT record
	-- Calls appropriate handler for record type
	set record_strings to {"form", "group", "html", "link", "picture", "rtf", "rtfd", "sheet", "txt", "xml", "record"}
	set record_handlers to {handle_form_rec, handle_group_rec, handle_source_rec, handle_link_rec, handle_media_rec, handle_text_rec, handle_text_rec, handle_sheet_rec, handle_text_rec, handle_source_rec, handle_form_rec}
	
	tell application "DEVONthink Pro"
		set type_str to type of rec as text
	end tell
	
	-- get handler for record type
	global rec_handler
	set rec_handler to handle_media_rec
	repeat with i from 1 to count of record_strings
		if item i of record_strings is type_str then
			set rec_handler to item i of record_handlers
			exit repeat
		end if
	end repeat
	
	rec_handler(rec, command_str, output_str)
end handle_record

on handle_input(command_str, output_str)
	-- Get selection from DT and handle as appropriate
	-- This supports selected text and one or more selected records.
	tell application "DEVONthink Pro"
		set rec_list to the selection
		set input_data to selected text of think window 1
		-- test if input_data is undefined
		try
			set jnk to input_data
		on error
			set input_data to ""
		end try
	end tell
	
	if rec_list is {} then error "Please select some text or at least one record"
	
	if input_data is not "" then
		perform_text_command(input_data, item 1 of rec_list, command_str, output_str)
	else
		repeat with rec in rec_list
			handle_record(rec, command_str, output_str)
		end repeat
	end if
end handle_input

-- *** Get Options ***
on get_command()
	--	Get command to execute from user.
	--	If history is enabled, the most recent history item is displayed, and a button to
	--    select a command form the History appears.
	if my enable_cmd_history then
		set cmd_str to last_cmd_in_history()
		set btn_list to {"Cancel", "History", "OK"}
	else
		set cmd_str to ""
		set btn_list to {"Cancel", "OK"}
	end if
	
	display dialog "Enter the command to filter input through:" buttons btn_list default button "OK" default answer cmd_str
	
	if my enable_cmd_history and button returned of result is "History" then
		-- show list of items in history
		set user_sel to get_cmd_in_history()
		-- user did not select a history item; recurse to get command.
		if user_sel is "" then return get_command()
	else
		copy text returned of result to user_sel
	end if
	
	return user_sel
end get_command

on get_output_option()
	-- Show list of output options to user
	--  These include Discard, New Record, Append Record, Replace Record, Append Comment,
	--  Send to Clipboard, and Send to DT log.
	
	set output_opts to {"Discard (/dev/null)", "Create New Text Record", "Append to Record Text", "Replace Record Text", "Import as File", "Append Comment of Record", "Send to Clipboard", "Send to DevonThink Log"}
	
	set output_strings to {"discard", "new", "append", "replace", "import", "comment", "clipboard", "log"}
	set user_sel to choose from list output_opts with prompt "Select output destination"
	repeat with i from 1 to count of output_opts
		if item 1 of user_sel = item i of output_opts then
			return item i of output_strings
		end if
	end repeat
	
	return item 1 of output_strings
end get_output_option



-- *** main ***
set command_str to get_command()

set output_str to get_output_option()

handle_input(command_str, output_str)

-- clipboard is the only output handler that has to be
-- dealt with after all records have been filtered.
if output_str is "clipboard" then
	tell application "System Events"
		set the clipboard to clipboard_str
	end tell
end if

-- if we got this far, command is worth saving
if enable_cmd_history then
	add_to_history(command_str)
end if

OK, I’m just about ready to start managing our internal data in DT :slight_smile:

–Eric

Thanks for the new scripts. Any feedback on my previous post?

Hi sjk,

I had similar thoughts once I got dt_record_info.sh going, after having tried to keep DT databases on two machines in sync. I haven’t made much headway, hence the silence :slight_smile:

Absolutely correct, there will have to be a ‘master list’ script. Replicants will have identical record IDs but different paths, so they could be handled as a special case (e.g. use a list of records with distinct record IDs to do the content synchronization, then synchronize all replicants).

Taking a step back, we are looking at writing a system to automate similar-but-different tasks:

  • synchronization of entire databases between two machines
  • synchronization of portions of databases on one or more machines
  • tracking changes to a database over time

This suggests the following components:

  • Master List. This takes options including the path(s) to list, whether to ignore directories excluded from classification, whether to include empty groups, and whether to make the paths in the output absolute (i.e. starting at /) or relative (i.e. starting at the specified paths). Output should be location + name, id, number of replicants, modified date, label, comments,state, and attached script (i.e. all meta information that may have changed without effecting the modification date). Obviously this will use STDOUT so that it can be run over ssh. NOTE: comments will have to be preprocessed with s/\n/\n/ as they will have newlines.

  • Export. This will take the output of Master List via STDIN (assume that a diff utility is run between them) and will generate a tarball of the records suitable for import into DT (after untarring). The tarball will have to include a directory (not for import) containing any scripts to be attached and a list of metadata changes to make (e.g. comments, state, replicants, etc). Options for this would include the name of a file to use instead of stdin, the name of the output file, an ‘ignore metadata’ flag, and the option to use uuencode over STDOUT instead of writing an output file. NOTE: links will be awkward to handle, as the files may not reside in the original locations (which themselves may not exist or be writable) on the destination (import) system.

  • Import. This script will handle the tarball produced by export. The options will include the name of the input file (or uuencoded data via STDIN) and the path to import the files into (defaulting to /). Metadata will have to be handled carefully: if a replicant doesn’t exist then create it, if script does not exist then copy it from the archive to where it is supposed to be, etc. Duplicates are just records with identical content, so they will be handled automatically.

  • Diff. This is going to be a tough one. The input will be two master lists (one via STDIN, the other by file or by running Master List on the current DB). The output will be a selection of lines from one of the lists (specified on the command line, e.g. -1 or -2) suitable for feeding to Export.

  • Sync. A controller script that invokes the previous scripts, over ssh if need be. Should be a no-brainer.

Obviously ‘Diff’ is going to be hard to get right. Diffing records purely by modification time is OK, but not ideal. For example, comments and state will change without the modification date changing. If a record has a different comment in each DB, but the modification date is the same, which version should be used? The notion of a ‘from’ database and a ‘to’ database would have to be employed, with the ‘to’ being overwritten by the data of ‘from’ in cases like these.

Then there is the merging issue. Given that this is an unsupported script written by end users, we can punt here, but it is worth considering the issues. If a record is modified in both databases, the one with the older date is going to be overwritten. A ‘date of last synchronization’ could be stored in the database to detect such files, with duplicates made of conflicting files before they are overwritten (and this, of course, could be controlled with a command-line option). If such support is added, ‘sync’ could actually be made two-way.

Well, those are my initial and somewhat lengthy thoughts on the matter. I think I’ve indentified enough of the issues and outlines enough of the features to start working on this.

Any ideas, while this is still in the planning stages?

–Eric

Still haven’t had a chance to fully digest what you’re proposing so this is just a quick acknowledgment. The synchronization aspect of it is certainly more ambitious than what I had in mind.

Well, I realized that if I make the synchronization sophisticated enough, then I can share entire groups between databases (e.g. between myself and coworkers, for documentation).

Here’s a quick Master List script:


#!/bin/sh
# dt_sync_list.sh: Produce list of database contents
# output is:
# DT_PATH|ID|NAME|LOCATION|MOD_DATE|NUM_REP|EXCLUDE|STATE_VISIBLE|STATE|LOCKING|ALIASES|ATTACHED_SCRIPT|LABEL|COMMENT

HELP_STR="Usage: $0 [-erx] [-d date] [-p path] [path ...]
	Options
	    -d date  Only list entries modified since 'date' (see Note)
	    -e       Include empty groups
	    -p path  Use DT database at 'path' instead of current database
	    -r       Use relative (to 'path' param) paths in output
	    -x       Ignore records excluded from classification
	Note: The date format must be recognized by Applescript's 'date' class.
	       Apple provides the following examples:
	      '7/25/53, 12:06 PM'; '8/9/50, 12:06'; '8/9/50, 17:06'; '7/16/70'; 
	      '12:06'; and 'Sunday, December 12, 1954 12:06 pm'."
	
DB_PATH=""
OPEN_DB="false"
MOD_DATE="current date"		# using "" as MOD_DATE causes the compile to fail
CHECK_DATE="false"
EXCLUDE="false"
EMPTY="false"
REL_PATH="false"

while getopts erx\?d:p: opt
do
	case "$opt"
	in
		d) MOD_DATE="date \"$OPTARG\"" CHECK_DATE="true";;
		e) EMPTY="true";;
		p) DB_PATH="$OPTARG" OPEN_DB="true";;
		r) REL_PATH="true";;
		x) EXCLUDE="true";;
		\?) echo "$HELP_STR";  exit 1;;
	esac
done


NUM_ARGS=`expr $# - $OPTIND + 1`
if [ $NUM_ARGS -eq 0 ]
then
	# List entire database
	export_paths=("/")
else
	shift $((OPTIND - 1))
	for (( i = 0 ; i < $NUM_ARGS ; i++ ))
	do
		export_paths[$i]="$1"
		shift
	done
fi

# -------------------------------------------------------------------------
function export_dt_group () {
	GROUP_PATH=$1
	if [ "/" = "$1" ]
	then
		ROOT_REC="(root of db)"
	else
		ROOT_REC="(get record at \"$1\" in db)"
	fi

	# Applescript to export group from database
	OUTPUT_STR=`osascript <<EOF
global nl
set nl to "\\\\\\\\n"	-- newline (escaped)
global dl
set dl to "|"		-- delimited (pipe)

on list_record(r, path_str)
	-- Returns a pipe-delimited listing of record terminated by a newline
	tell application "DEVONthink Pro"
	set rec to path_str & name of r & dl & id of r as text & dl
	set rec to rec & name of r & dl & location of r & dl
	set rec to rec & modification date of r & dl
	set rec to rec & number of replicants of r as text & dl
	set rec to rec & (exclude from classification of r) as text & dl
	set rec to rec & state visibility of r as text & dl
	set rec to rec & state of r as text & dl & locking of r as text & dl
	set rec to rec & aliases of r & dl & attached script of r & dl

	-- need to remove emedded returns from comment
	copy comment of r to the_cmt
	set AppleScript's text item delimiters to return
	set cmt_list to text items of the_cmt
	set AppleScript's text item delimiters to ""
	set cmt_str to ""
	repeat with cmt in cmt_list
		-- all those backslashes just to get '\' 'n' in the comment
		set cmt_str to cmt_str & cmt & "\\\\\\\\\\\\\n"
	end repeat

	set rec to rec & label of r as text & dl & cmt_str & nl
	end tell
	return rec
end list_record

on list_group(r, path_str)
	-- Returns record listing for group record, followed by a
	-- recursive (depth-first) listing of all records in the group.

	-- get listing for group record
	set out_str to list_record(r, path_str)
    
    	tell application "DEVONthink Pro"
        	set g_children to children of r
			set path_str to path_str & name of r & "/"
    	end tell

   	repeat with c in g_children
		-- recurse on record
		set out_str to out_str & my handle_rec(c, path_str)
	end repeat

	return out_str
end list_group

on handle_rec(r, path_str)
	-- Return listing for record and all child records, unless the
	-- record fails the Date, Exclusion, or Empty check.

	-- perform date check (TODO: verify group mod_date is accurate)
	if $CHECK_DATE then
		set min_date to $MOD_DATE

		tell application "DEVONthink Pro"
			set rec_date to date of r
			set loc to location of r	-- cannot get date of root record
		end tell
		if loc is not "" and rec_date <= min_date then 
			return ""	-- record not modified recently
		end if
	end if

	-- perform exclusion check
	if $EXCLUDE then
		tell application "DEVONthink Pro"
			set excl to exclude from classification of r
		end tell
		if excl then
			return ""	-- record is excluded from output
		end if
	end if


	tell application "DEVONthink Pro"
		set num to number of children of r
		set k to kind of r
	end tell

	if num > 0 or (k is "group" and $EMPTY) then
		set out_str to list_group(r, path_str)
	else if k is not "group" then
		set out_str to list_record(r, path_str)
	else
		set out_str to ""
	end if
	
	return out_str
end handle_rec

-- *** main ***
tell application "DEVONthink Pro"
	-- get alternate DB
	if $OPEN_DB then
		try
			set db to open database "$DB_PATH"
			get name of db
		on error
        	return "ERROR: Invalid database path $DB_PATH" & nl
		end try
	else
		set db to current database
	end if

	-- get starting point
	copy $ROOT_REC to root_rec

	-- set absolute or relative path
	if $REL_PATH then
		set path_str to ""
	else
		set path_str to location of root_rec
	end if
end tell

try
    -- verify that path exists: nested try is an attempt at 'if defined(r)'
    get root_rec
    try
		set result to handle_rec(root_rec, path_str)
    on error msg
        set result to "ERROR: " & msg & nl
    end try
on error
    set result to "ERROR: Invalid path" & nl
end try


EOF`
}
# -------------------------------------------------------------------------

# export each path specified on the command line
for (( i = 0 ; i < ${#export_paths[@]} ; i++ ))
do
	export_dt_group "${export_paths[$i]}"
	echo  -ne "$OUTPUT_STR"
done

exit

The main selling points are that it lists the salient (as far as sync goes) details of all records, including groups, and that it allows limiting options such as record date and exclusion from classification.

I should note that this takes FOREVER when run on / of my main database (2K groups), but performs reasonably well on any sub-groups of /. I think it is a couple of problem groups (sheets containing thousands of records) that slow it down, but I haven’t spent any time tracking it down.

The next thing to do would be the diff utility, then the export, followed by the import. Not sure when I’ll have time to tackle those though :slight_smile: