Stuck points in developing an improved workflow (new user)

I came to a point earlier this year where I realized I am spending SO much time navigating through various folders in the Finder and forget about key resources buried in various places. I also came to realize how terrible Microsoft Word is (nothing stands the test of time, terrible with long documents, etc.) Since, I’ve been on a mission to develop effective workflows to save time and stay organized. I’ve decided to transition to a plain text note-taking strategy (no more Word). I’ve downloaded and been trying to understand how several different apps can help me recreate my electronic workflow and storage system. I’m looking for feedback about a few stuck points I have been unable to resolve on my own so far.

Important caveat: I am not a coder or programmer and don’t know how to script! I’ve figured out Markdown, but that’s pretty much it. (I’m an academic in psychology.)

Set up: Macbook pro (upgraded memory and SSD, dual external monitors, treated as desktop); ipad pro (this is what I take on the go instead my macbook, which stays stationary); iphone

Apps that I’ve invested some time/money on and might be useful:
DEVONthink Pro Office & DEVONthink ToGo: I purchased but haven’t set up yet, but learned the basic, waiting for Mojave release due to issues with Mojave beta
Scrivener: I’m already obsessed!
nvAlt + 1writer: plain text note-taking on ipad and mac; synced on icloud
Keyboard maestro: Free trial downloaded but am not sure if I have the coding skills necessary to make that a worthwhile purchase.
Zotero: This was the one efficient part of my workflow that was already established. I’m not willing to give it up either. I use Zotfile to auto-rename and move all of my research material stored in Zotero to specified finder folders on my mac. I have nearly 4000 PDFs and other items indexed and referenced in Zotero but stored externally in Finder.

STUCK POINT 1:

I’m a researcher, so both creating and consuming published research (usually journal articles) is a huge part of my job and my day-to-day. I often take notes on a given article. Now all of those notes (for the most part) – which were scattered all around my hard drive – have been converted to plain text files and compiled in a single folder called “Literature notes” (about 600 items total; this is kept separate from my nvALT note database). I realized my system has been so terrible that I found many duplicate annotated PDF articles and even multiple – different – ‘note’ files for the same article. Terrible, terrible. Anyway, i’ve cleaned things up, now I have all of my literature (Zotero database) stored in a set of folders defined by ‘item type’ (i.e. journal article folder contains the bulk of the data base – almost 3000 items total. So I have literature in a small set of folders and then I have all of my literature notes stored in a separate folder. I’m concerned about combining them given the number of items. i’m afraid Finder will lag, as it’s already lagging a tiny bit with a 3000-item folder. All of my ‘literature notes’ text files have the same exact name as the corresponding PDF (or whatever type of document indexed in Zotero).

e.g. I’m reading an article. the PDF is titled “Clevenger_Schrepf et al_2013_Cancer.pdf”. I create a new text file to take notes on this article, and save it as “Clevenger_Schrepf et al_2013_Cancer.pdf.txt”. Obviously with batch renaming, this file naming approach can be modified if needed.

This is the ideal outcome I would like to develop:

  • I want to be able to merely glance at a ‘literature’ item in Zotero (I’m calling everything in Zotero ‘literature’, for clarity) or in its location in the Finder and know very quickly whether or not I have taken a note on it – that is, does it have a separate text file? (note: I’m not interested in storing notes in PDF files themselves). Right now I do this by navigating to the ‘literature notes’ folder – for which the ‘search’ function is so slow that it’s useless – and scroll to find it by sight. The most obvious option is store them in the same folder, sort by name, and that’s all it takes. But with the volume of documents i’m working with, this will slow me down and makes me worried, as long as I rely on the Finder. I can’t use the Finder to search through the folder as is because it’s too slow, so that’s also why I don’t think that would work.
    My ideal workflow for this would look like:
  • For new items: Download a new item in Zotero. Automatic renaming and moving to designated location in Finder. Open PDF from Zotero. Use shortcut to automatically create and pull up new plain text file, titled with the same title as the new PDF document.
  • For existing items: Open PDF from Zotero – automatically also pull up the corresponding plain text file – if it exists already. If it doesn’t, automatically create and save new one, using same naming conventions.

One idea is to make aliases for all notes and store them together with the source PDF so at least one of my outcomes is met. but that doesn’t solve the automatic linking between PDF and note. What about using DEVONthink and/or keyboard maestro? Again, i’m new to both programs, but eager to learn.

STUCK POINT 2:

I have regular weekly meetings with coworkers, during which I take notes about several different projects that we discussed in a single meeting. On my mac, these projects are in separate folders. I’m left, though, with all of my meeting notes (text files) having information belonging to several different projects, stored in several different places.

My ideal workflow here would be:

  • As I’m taking notes during a meeting, be able to somehow flag/mark different selections of text (within a single text file while editing in 1Writer on Ipad) by topic as I’m typing.
  • Have an automated process set up where only the select text flagged with topic indicator gets COPIED to a different location – ideally another specified text file stored somewhere else (like in its designated project folder).
    for example: one of my meeting notes might look like this:

project x:

  • note1
  • note2

project y:

  • note3
  • note4

I want to find a way to automatically move and file text for project x to a designated ‘project x’ file and storage location. Same for project y.

Help!!!

I’m a little confused here. You are talking about using and working in Zotero and also in 1Writer.
It’s unclear how you are viewing our products in the situation.
What have you actually tried in DEVONthink or DEVONthink To Go?

Personally I would modify your workflow and index this literature folder in DevonThink, then you have all of DevonThink’s scripting and search tools at your disposal - whilst not breaking any of your existing setup. You could even use a script like the following to create new notes for your files with the same name as the reference material


tell application id "DNtp"
	try
		set this_selection to the selection
		if this_selection is {} then error "Please select some contents."
		
		repeat with this_item in this_selection
			set theName to name of this_item
			set theRecord to create record with {name:theName, type:markdown, content:theName} in the current group
		end repeat
	end try
end tell

I apologize for the confusion - I’m new to DTP and to writing on forums like this in general. I have been getting to know DTP and trying to figure out how I will use it. Yes, I would love to use DTP to fix my stuck points, but I’m struggling to understand just how i would do that. I’ve been trying to learn and understand the many ways DTP can be used and its functions. One specific concern I had with using DTP to create and link all notes is portability and syncing. Ultimately, I think I’m hung up on how to create a system in which I don’t have to always rely on DTP to access and create and update my documents. Please keep in mind that I have no experience working with any program like this before or even thinking about my own process at this level. What’s obvious I’m sure to you is not to me, so any obvious ideas or thoughts would be much appreciated.

I so appreciate your thought, and this is exactly the sort of response i was hoping for – thank you! But I need to first say that you’re already way over my head. I can’t read/write – is that Applescript? I’m feeling like this is a big part of why I’m stuck - what I’m trying to do might just require skills that I just don’t have. I’d love to learn, if you have advice, by the way - it’s a daunting thing and hard to know where to start.

re: what I could understand from your suggestion - am i correct in that if I create a new note in DTP, that file is automatically kept within the DTP database, and not easily accessible outside of DTP or DTP To go? To me, that’s a big commitment that I worry I won’t be able to be consistent with. Synchronization/access on the go – or at the very least some system that prevents duplicates and conflicted copies – is important for me. I’m also not the type to reliably and regularly export various files periodically to the finder to make sure everything’s in the same place – it’s just not realistic, I won’t do it. And I’ll end up confused with multiple copies in multiple places from when I couldn’t access DTP or DTP to go in the moment, etc. That’s my fear. Tell me there’s some way around this?

I would suggest you read the manual section on indexing (titled “Index files on your hard disk”). However as a summary: Nothing will be exclusively in the DevonThink DB with this, it will be where it has always been, on your hard drive (well, hopefully an SSD) - and new files you create will also be there.

The script is indeed AppleScript, and it creates a new note with the exact same name as the file(s) currently selected in DevonThink - and in the same place too.

No worries :smiley:

So what do you envision DEVONthink being used for? Tools are great, if they’re needed for a specific purpose. You need to being this journey by figuring out the answer to this. Seriously. This will certainly help you find direction.

I can certainly offer advice in relation to your first question (re: Zotero), and Rosemary’s advice about indexing your literature folder is a good start.

Basically, my advice is to use Zotero simply to store citations and use Devonthink for everything else (accessing the PDFs, labelling them, taking notes on them, etc).

I have a folder on my computer called ‘Bookshelf’ which contains all the PDFs of journal articles I have collected from around the web. The Zotero extension downloads the PDF and associated metadata into Zotero Standalone. Zotfile then automatically renames the PDFs (since I’m a historian I always like to have the year of publication as the first part of the file name!) and moves them to the Bookshelf folder.

I then index this folder with Devonthink so I can access and search all my PDFs from within my database. I then use the replicate function to place specific PDFs into particular subject-specific folders. Unread articles are labelled red.

For note-taking, I use the built-in annotation template (Data > New from Template > Annotation). This creates a rich text document but you can download a markdown annotation template if plain text’s your thing. I usually move the annotations to a separate Notes database, but they remain linked to the original PDF. To see all your notes at a glance, you could create a smart group searching your database for any file where Name matches (Annotation).

As you can see, Devonthink is incredibly powerful but for workflow purposes I find it’s better to separate out the citation functions of Zotero with the information-management functions of Devonthink. I’ve largely stopped using a dedicated note-taking app because it adds to the complexity and Devonthink’s editor is more than adequate for most purposes. If you want added functionality you could always use a separate notetaking app and save the fiel to Devonthink (I haven’t used iAWriter, but I’ve sometimes used Byword for this purpose).

1 Like

Also, any PDF you’ve made notes on using the annotation template will have an x-devonthink- somethingorother in the URL field, so this is another easy way to see at a glance what documents you’ve made notes on, unless you’ve deliberately linked to something else.

Thanks so much for following up. Right – but since we’re advised against placing DevonThink DB data on iCloud, this restricts access to all my files to just that one device. I rely heavily on access to all of my documents via iCloud and use 3 devices to access files regularly. I have read what you suggested about indexing and importing as I’ve been learning about the app and how it can help improve my workflow. Indexing my zotero database of journal article PDFs, etc., works fine for me. What I have gathered, though, is that if I were to regularly create new files within DT – for example, annotations/‘literature notes’ (a snag in my workflow that i’m trying to improve) – because they are by default stored on my SSD but not in iCloud, there’s not an easy way to automate whatever needs to be done to sync my content across all devices. Am I understanding right?

Thank you so much for your thoughts - this is so helpful! Sounds like we have similar tasks using these apps. I also use Zotfile to rename, link, and move all my files – I don’t rely on Zotero to store any actual data other than citations. I just find Zotero so much better than Finder to work with my sources, so I’m ‘in it’ a lot. Anyway, totally agree and am with you up on your Zotero procedure.

It sounds like you index sources but create annotations within Devonthink. This is basically what I found to be the logical resolution to my issue too – index literature, create notes in Devonthink using the templates, linking, etc. But I don’t think it’s a realistic solution for me, based on my understanding that notes (for example) created within Devonthink are automatically (and advised to be) stored on the local SSD, not on the iCloud. This means I can’t sync with my iPad and iPhone. I rely heavily on icloud syncing and access (I use my iPad rather than macbook on the go). Does this sound rightt? Do you have any ideas about getting around it?

Also had to say I so appreciate this point!! I really want to be able to glance and see, so duly noted!

I have had no problems at all syncing notes with Devonthink to Go on my iPhone and iPad, and I use iCloud as the sync mechanism. Note that this isn’t the same as iCloud Drive, which is possibly where the confusion comes from. (I won’t go in to details but suffice to say you should NEVER place your databases directly in a cloud synced folder - the options for iCloud and Dropbox you see in the sync preferences panes use these sync systems as a kind of back-end). Thus keep your notes within Devonthink itself and allow Devonthink do the syncing.

As long as you select iCloud and not iCloud Drive, sync will work well between your Apple devices. I replaced Dropbox with iCloud as the sync system a few months ago and I’ve been impressed with its reliability and speed. Plus I can get rid of Dropbox as a subscription!

In sum, this is what I do. You might find something similar works for you.

  • On the Mac, all my Devonthink databases I stored in my home directory, somewhere like ~/techyhistorian/databases. NEVER EVER EVER put your actual databases in your iCloud folder or Dropbox etc.

  • My Bookshelf folder, which I have Zotfile move all my PDFs, is located at ~/techyhistorian/documents/. This doesn’t actually need to be in your iCloud folder since Devonthink will sychronise indexed documents (see below).

  • Bookshelf folder is indexed by Devonthink.

  • Notes on these PDFs are created within Devonthink itself and are not stored externally on my computer.

  • In the Sync preferences pane, select iCloud (NOT iCloud Drive), enter any credentials, and the databases you want to sync. In my case this would be the ‘Sources’ database where I have all my PDFs, and the ‘Notes’ database where I keep all my notes.

  • On your iOS devices in DTTG, you can then set up sync the other way (look at the manual for the exact steps you need to take).

Hope this all makes sense!

2 Likes

Hi,

This has been an informative thread that is right up my alley. Thus I am continuing the thread rather than creating a new topic.

I am, not uncommon to the users here, looking at DT to better access an extensive archive of several thousand PDFs on what amounts to be 3 high-level topics. I’m on a Mac laptop and have the bulk of the material (the thousands of PDFs) are in a Finder hierarchy on DropBox. There are some notes (2-3k) in Evernote and, of course, there is email.

Major issue for me: storing my research (and all other documents) in the cloud. My “library” of PDFs is on DropBox, as are my writing projects (I use Scrivener but my various editors use Word, so I have both files). However, I have some documents (mostly images and others) on iCloud Drive. (And no, no folders or files are duplicated or otherwise stored on both.) The reasons for this arrangement are a) no-thought automatic backup (my laptop, previous TimeMachine or other backups could disappear, the OS could become corrupted, etc but my data is immediately available), b) iPhone access to these files. When there is a system failure or hardware upgrade, when (re)building my machine I have no desire to bring over the past OS and prefer to start from scratch with fresh installs, an old habit going back to my earlier days on DOS and reinforced with Windows 286 & 386 and on. There is no reason of substance why I use both except I was on DropBox first and then iCould Drive became available. I can see transitioning wholly to iCloud, but I don’t think this is germane, but maybe it is and someone will tell me. The much of the library of PDFs has been built up through weeks spent at several presidential and other libraries around the country, different national archives locations, as well as PDFs acquired online.

It’s pretty clear, without ambiguity, that DT does not permit storing the db on a cloud drive. This is a concern for me as it appeared I was moving my PDF archives (Finder folders and files) into DT. However, this thread suggests this is not required or possibly intended and I misunderstood something.

My questions are (and feel free to let me know if I’m asking the wrong questions because I misunderstood something):

  1. Is it ‘good’ or ‘best’ practice to maintain PDFs outside of the DT database? Clearly, the Evernote and email and organic notes are maintained in DT. And if so, how does this work because it appears PDFs, and not mere pointers, are copied into DT? Should I not care about the duplication of PDFs as I’m not really, as of now perhaps naively, concerned with the size of the DT databases?

  2. On the Mac in Settings > iCloud > iCloud Drive, there is DevonThink Pro Office (which I am still evaluating). What is this syncing if not the database?

  3. Not related to the backup discussion but related to PDFs, how can I access and use as a pointer or reference point, if not search, bookmarks I created in PDFs?

Thank you for the thread and for the help with my questions.

Regards,
Matt

There is no “best practice” on this, as it’s relative to the need. I personally have no need to put any of my data in the cloud, PDFs or not. Does that make it a “best practice”? For me, it absolutely does. If I was in a situation where I needed to collaborate on some PDFs, then putting them in my local Dropbox folder and sharing a common Dropbox account would be a “best practice”. Etc.

It is Syncing raw DEVONthink-specific data for use with other devices running DEVONthink or DEVONthink To Go 2. (Sync is also not a backup.)

There are two methods of putting data in a DEVONthink database.

  1. Importing copies files into the internal structure of the database. This creates a portable, self-contained database that can be moved as a single file. This is the default behavior of DEVONthink…
  2. Indexing creates links to files outside the database. This allows more direct access to the files in the Finder by other applications or people. (A common example is people using a cloud service like Dropbox and indexing folders in the local Dropbox folders.)
    It is best used on fairly static locations, ie. you’re not moving the folders around in the Finder and renaming them. However, if you do index a location, you can update it in the database and it will change to reflect the changes in the Finder.
    Note: Indexing should be approached thoughtfully, and a search on the Forums is advisable, paying attention to posts by Greg Jones, as he uses indexing almost exclusively.

This in not entirely accurate. What about when the network is down or unavailable, or servers slow / unresponsive?
I can travel less then 10 minutes from a small metropolitan area and have little to no network connection. Also, cloud services aren’t bulletproof and network conditions greatly affect access to Internet resources. So I can easily get in a situation where I have no network access. On the other hand, I have never had a Time Machine drive “disappear”. :wink:

Jim,

Thank you for your timely reply. This helps clarify my thinking, especially with regards to the issue of moving or copying files into DT.

I may have been unclear with my question #3 about bookmarks in PDFs. This is specifically about PDF bookmarks made in the PDF using Acrobat. I have created many bookmarks in many of the PDFs to the point I change the default opening behavior of each PDF I review to “Bookmarks Panel and Page.” I cannot see how DT accesses/includes in searches these bookmarks. Am I missing something?

To clarify my point about the cloud, whenever the network is down, or more likely when I’m on a plane, in a coffee shop, or otherwise where I have (or choose to have) no network access, the cloud drives (DropBox and iCloud) are mirrored on my laptop and thus to me there is no interruption in access. Changes are of course synchronized the next time I’m online (when I’m in the hotel, personal hotspot in the taxi or site, at home, etc). These files are not shared with others (unless I copy them to a specific folder to be shared), my concerns about conflicting versions (the other problem with the cloud) are minor as the issue for me, even when I’m touching files on the iPhone, is exceptionally rare. And in the case my device (laptop and/or phone) disappears (theft, fire, accident) or becomes disabled (corruption, failure), being without direct access to these files until I acquire a new device (including repurposing an old device or taking over my wife’s desktop in the gap) is not fatal and often not too painful (if relaxing) if measured in days.

Thank you again.

Matt

No problem.

Regarding Acrobat Bookmarks, they are not supported in DEVONthink at this time though it may be part of a future release.

sounds interesting - the “standard annotation template” cannot be edited on DT2Go - where can I download the template? the template download area doesn’t show a ‘different’ annotation template.

Thank you, Klaus

I never knew that. On reflection, it makes perfect sense, though.